Recently by Aaron McCurry
While I'm working on getting my code into lucene. I have patched 3.0.0 and 2.9.1 with my low memory patch.
By default the small memory footprint is enabled to change back to the default implementation set the following system property.
-Dorg.apache.lucene.index.TermInfosReader=default
Have fun! If you have any problems or questions please let me know or add to this LUCENE-2205.
Lucene gives users the ability to search massive amounts of data in a very short amount of time. However allowing users to page through the entire result set of their search can be difficult and risky depending on how many users are performing searches and how many of those users are paging through 100's if not 1,000's of hits per page.
The Simple Example:
The More Advanced Example:
Problem Scenario:
- Each of your indexes contains 100,000,000's documents.
- You have 500 users on your system actively performing searches.
- You have 100 search results per page.
- And, your typical user pages through the first 10 pages of results. (Normal occurrence on some systems)
So for the 10th page you will have to collect 1,000 hits, at a cost of a float plus an int plus some object overhead per hit. So let's say 20 bytes per hit. So you have 500 users * 1,000 hits * 20 bytes = 1,000,000 bytes or 1M. Easy, no problem, right?
Well what if you also give the users an easy way to move to the end of the result set. Hmm... Well for a result set size of 10,000 it's no big deal. But what if you hand out result sets in the order of a 1,000,000 or even 10,000,000.
At this point you really just want to prevent the system from running out of memory. Because if you have 25 users getting 10,000,000 results each and they all click last page at the same time. That's going to cost you 5 Gig of heap! At least. Some might say that it won't ever happen, but in my experience, if it can happen, it will.
So I created a Paging Hit Collector, that windows the hits to the users. It's uses the last hit collected from the previous search pass, to feed the next search pass. So yes if a user clicks the last page, it might perform multiple searches but, the system won't run out of memory.
The user's will get there answer eventually, and if your system gives them some feedback as it searches and pages, they will probably sit and wait for it to come back. Instead of giving up and hitting cancel and search and cancel and search, and making the system worse and worse.
IndexSearcher searcher = new IndexSearcher(reader);
TermQuery query = new TermQuery(new Term("f1", "value")); IterablePaging paging = new IterablePaging(searcher, query, 100);
for (ScoreDoc sd : paging.skipTo(90)) { System.out.println("doc id [" + sd.doc + "] " + "score [" + sd.score + "]"); }
IndexSearcher searcher = new IndexSearcher(reader);
TotalHitsRef totalHitsRef = new TotalHitsRef(); ProgressRef progressRef = new ProgressRef();
TermQuery query = new TermQuery(new Term("f1", "value")); IterablePaging paging = new IterablePaging(searcher, query, 100);
for (ScoreDoc sd : paging.skipTo(90). gather(20). totalHits(totalHitsRef). progress(progressRef)) {
System.out.println("time [" + progressRef.queryTime() + "] " + "total hits [" + totalHitsRef.totalHits() + "] " + "searches [" + progressRef.searchesPerformed() + "] " + "position [" + progressRef.currentHitPosition() + "] " + "doc id [" + sd.doc + "] " + "score [" + sd.score + "]"); }
Here's a link to the code LUCENE-2215.
I've been using Lucene for the better part of 2 years, from initial playing around, to prototyping to production application. It's an impressive library and it has come along way in the past couple of years.
When I first started playing around with it the version was 2.1 and the search times were so much faster than what we were trying to use at the time (Oracle Text). The first test was indexing a monster dataset and searching it quickly. It passed with flying colors!
Next was to add in record level access control. Easy and extremely fast.
Next was to add in all the other data needed for our application. That was a little bit harder, considering that we have close to 150 fields in our index and well into the billion record range (growing everyday).
The problem was that we needed more memory and there was no extra money for any more servers (or upgrades). So there we were, stuck. So I decided to start poking around using visualvm to see if there were any places in our application or in Lucene to save some memory.
We had already disabled norms on all our fields (we really didn't need norms for our data nor did we have the resources). Took a long look at all our fields that we were indexing to see if there were any we didn't need, but we really did need them all. Then I stumbled across the TermInfosReader class in Lucene.
This is where Lucene really gets it speed, but also uses quite a bit memory to do it. And this is where I wrote my first Lucene patch.
In TermInfosReader there is a bunch stuff but the big memory hogs are in three arrays.
- Terms[]
- TermInfos[]
- long[]
Basically Lucene does a binary search across the Terms array (that by default contains every 128th Term in the index) with a given Term to find where on disk the exact Term needed lives. There's a little bit more going on in the class than that, but that's basically what it's doing.
So, I started this patch with the need to save memory. So how in the world do you do that in java when everything is already in basic arrays and everything is needed in memory. Well you have to save it another way, references. References are a hidden cost in Java, every single reference in 32-bit JVM costs you 4 bytes, and 64-bit JVM it's 8 (assuming that you don't have compressed references).
Let's count the references.
- Terms[] length * 3, 1 reference for the Term and 2 references for the two Strings inside the Term
- TermInfo[] length * 1
- long[] = 1 reference total
So, let's talk numbers. If you have a billion terms in your index, that's 125 MB (1,000,000,000 / 128 * (3 + 1 references) * 4 bytes for every ref) bytes of memory for the references. In a 64-bit JVM that doubled 250 MB. Not to mention the object overhead for every one of those Term and TermInfos objects. Wow that's a lot!
So I decided to remove nearly all of those references by using a byte array and an int array as an offset index.
The results were impressive!
Given an index of 6.2 GB size 1,010,000 number of documents with 179,822,683 number of terms the default implementation uses 292,235,512 bytes to just get the index usable.
My no-ref implementation of the same index uses only 49,849,744 bytes get the index usable. That a 17% of the original size, that's an 83% savings!
And the best part is, that it loads the segments faster into memory. So those real-time updates will be online faster. The run-time performance is slightly faster as well. But the huge performance saving is in garbage collection. Over 7 times faster for full GC's on my Macbook Pro. Wow!
I think that the results speak for themselves, and I hope that the Lucene folks will accept my patch. That way I won't have to continue patching each version after the fact. Also removing references can be great, but the code required to do it, and maintain the same level of performance, is ugly! So don't try this at home!
At the beginning of last month I started prototyping various solutions for a customer using HBase. However I found myself writing tons of code to perform some fairly simple tasks. So I set out to simply my HBase code and ended up writing a Java HBase DSL. It's still fairly rough around the edges but it does allow the use of standard Java types and it's extensible.
Simple Put and Get Example
Direct HBase API:
Scanner Example
Direct HBase API:
See the unit tests, for more examples.
Simple Put and Get Example
Direct HBase API:
public class PutAndGet {
public static void main(String[] args) throws IOException {
HTable hTable = new HTable("test");
byte[] rowId = Bytes.toBytes("abcd");
byte[] famA = Bytes.toBytes("famA");
byte[] col1 = Bytes.toBytes("col1");
Put put = new Put(rowId).
add(famA, col1, Bytes.toBytes("hello world!"));
hTable.put(put);
Get get = new Get(rowId);
Result result = hTable.get(get);
byte[] value = result.getValue(famA, col1);
System.out.println(Bytes.toString(value));
}
}
HBase-dsl API:public class PutAndGetWithDsl {
public static void main(String[] args) throws IOException {
HBase<QueryOps, String> hBase = new HBase<QueryOps<String>, String>(String.class);
hBase.save("test").
row("abcd").
family("famA").
col("col1", "hello world!");
String value = hBase.fetch("test").
row("abcd").
family("famA").
value("col1", String.class)
System.out.println(value);
}
}
Now this is where the dsl becomes more powerful!Scanner Example
Direct HBase API:
public class Scanner {
public static void main(String[] args) throws IOException {
byte[] famA = Bytes.toBytes("famA");
byte[] col1 = Bytes.toBytes("col1");
HTable hTable = new HTable("test");
Scan scan = new Scan(Bytes.toBytes("a"), Bytes.toBytes("z"));
scan.addColumn(famA, col1);
SingleColumnValueFilter singleColumnValueFilterA = new SingleColumnValueFilter(
famA, col1, CompareOp.EQUAL, Bytes.toBytes("hello world!"));
singleColumnValueFilterA.setFilterIfMissing(true);
SingleColumnValueFilter singleColumnValueFilterB = new SingleColumnValueFilter(
famA, col1, CompareOp.EQUAL, Bytes.toBytes("hello hbase!"));
singleColumnValueFilterB.setFilterIfMissing(true);
FilterList filter = new FilterList(Operator.MUST_PASS_ONE, Arrays
.asList((Filter) singleColumnValueFilterA,
singleColumnValueFilterB));
scan.setFilter(filter);
ResultScanner scanner = hTable.getScanner(scan);
for (Result result : scanner) {
System.out.println(Bytes.toString(result.getValue(famA, col1)));
}
}
}
HBase-dsl API:public class ScannerWithDsl {
public static void main(String[] args) throws IOException {
HBase<QueryOps, String> hBase = new HBase<QueryOps<String>, String>(String.class);
hBase.scan("test","a","z").
select().
family("famA").
col("col1").
where().
family("famA").
col("col1").eq("hello world!","hello hbase!").
foreach(new ForEach() {
@Override
public void process(Row row) {
System.out.println(row.value("famA", "col1", String.class));
}
});
}
}
See the unit tests, for more examples.
Go is a new experimental language that has been released by Google that is suppose to have nearly the speed of C while executing, without a lot of the normal headaches of C/C++ coding. For example Go has garbage collection, dynamic types, reflections, etc. see Google Experiments With A New Language, Go for a good overview. So after reading a few things about it, I wanted to try it out, and here is how I got it working on my 64-bit OSX machine.
It's pretty basic, all I did was follow the instructions here.
First I added:
Run:
Note: I had to install the Xcode.mpkg from the Optional Installs on my Snow Leopard DVD to get gcc-4.3 installed.
Next:
And that's it! Time to make a hello world program.
Create hello.go:
It's pretty basic, all I did was follow the instructions here.
First I added:
export GOROOT=$HOME/goto ~/.bash_profile
export GOOS=darwin
export GOARCH=amd64
export GOBIN=$HOME/gobin
export PATH=$PATH:$GOBIN
Run:
. ~/.bash_profileNext:
sudo easy_install mercurialThis took a while for me to pull mercurial down, maybe their server was having problems.
Note: I had to install the Xcode.mpkg from the Optional Installs on my Snow Leopard DVD to get gcc-4.3 installed.
Next:
hg clone -r release https://go.googlecode.com/hg/ $GOROOTNext:
cd $GOROOT/srcNote: One of the http tests goes out and touches google.com, I had to drop the firewall while building everything. After the build and tests were complete I turned it back on.
./all.bash
And that's it! Time to make a hello world program.
Create hello.go:
package mainCompile and Link:
import fmt "fmt"
func main() {
fmt.Printf("hello world!\n");
}
6g hello.goRun:
6l hello.6
./6.outOutput:
hello world!I am very excited about Go can't wait to write something real with it!
