Stalwarts of Tech – An Interview with Peter Lawrey – Chronicle, Stackoverflow and Performance
About this series
This is the next interview in a regular series of interviews with stalwarts of the technology industry. We wanted to highlight many of the unsung heroes of the technology industry, the people and projects that have made huge impacts in our lives as developers and technologists!
Peter Lawrey – Chronicle, Stackoverflow & Performance
We’re really pleased to have Peter Lawrey, CEO at OpenHFT, a performance guru, longtime London Java Community member and not to mentioned he’s probably answered one of your questions as a top 5 contributor to Java Stackoverflow! Peter is well know for several reasons, most recently his Chronicle library for high performance messaging in Java. So without further ado, let’s find out about Peter’s contributions, his history with performance tuning, motivations behind Chronicle and his thoughts on Java and developers today!
1. Would you like to introduce yourself Peter?
No. I don’t like introducing myself :-). I have to sell my services to get work as a freelancer, otherwise I am happy to just introduce myself as Peter.
I am a Server Side Core Java Developer who has been working on low latency trading systems for 5 years, with 22 years experience in IT. I have worked with investment banks, trading houses, and hedge funds. I am the lead developer of OpenHFT and our lead product is Chronicle which is designed to simplify fast data in Java. It can synchronously persist data and share it with other JVMs in around a micro-second!
2. You’re famously a top 5 Stackoverflow contributor for Java, where do you find the time?!
I think the interobang or ‽ is the coolest piece of punctuation. I tend to use it as !? quite a bit.
I also like taking questions out of context. Perhaps this is why I don’t mind answering the naive "mis-fired" questions no one else will answer.
Stackoverflow.com is where I learn a lot about Java by answering questions, especially the obscure points! It is also something I can do when I need a distraction. I am on there nearly every day.
3. Which skills do you see need the most improvement in developers?
Many developers don’t use the available tools enough. Most use a fraction of what their IDE can do for them, some don’t use an IDE at all. The IDE can highlight many common errors with static code analysis, and yet developers will spend hours debugging their code and when they solve it I point out to them, see that yellow outline of the code, that is what your IDE has been trying to tell you all along.
Every developer should know how to use a debugger effectively and they should be working in an environment where they can use their debugger effectively. In many environments, the application was never designed to be run regularly or realistically on the developer’s work system. This is a serious mistake IMHO as the development lifecycle of an application goes up dramatically if developers cannot test their own code with the click of a button.
Every developer should be familiar with a performance profiler. A profiler is useful even if you don’t care about performance, as they give you a different view into your application. I often find non-performance related bugs by looking in a profiler. e.g. there is 100 copies of a singleton, not a performance issue, all the tests pass, but still a plain bug. You shouldn’t just know how to run a profiler, you should learn how to understand what it is telling you. Many times I have seen a report which to me couldn’t be more obvious and the client will say, "see it, doesn’t show anything", except the top ten things you should optimise first ;-).
Using the right tools can make such a difference to the quality and expectations of what you intend to deliver as a project. For GC tuning issues, Censum can save not only a lot of time, it raises the expectations for the project. A common mistake I see Java developers making is; "It pauses, it must be the GC", like that explains everything. In truth there is often something simple you can do about it, like remove command line options which are not helping, or it might not be the GC at all.
4. Which skill do you feel students have lost?
One skill I have seen get lazier is the ‘art’ of cheating at homework. There was a time when students would go to great effort to find another students work and copy it. I used to think laugh at such people, partly because I thought it would have been easier to just do the work and you might learn something.
Today it couldn’t be easier to find pieces of code and stick them together, but some students can’t even do that. They post parts of question in the hope someone will do the thinking for them and often:
- They could have googled the answer with less effort.
- There is no way the marker or the assignment is going to believe the student wrote the answer, or often.
- There is no way the student has done enough research to be able to understand the answer.
5. How many days does it take to master Java?
There is a common misconception that once you have learnt the syntax for Java and written a few programs you can be an expert in Java. "Now I have been learnt Java over the last few weeks, which language should I learn next" or "Learn Java in 21 days". On the flip side of that there is a common problem labelled as the ’imposter syndrome’ where a developer realises they don’t know as much as they think others do, and live in fear they are going to be found out one day and lose their jobs.
The reality is you are constantly learning, you could be programming Java for tens years, and only know really well a fraction of the Java ecosystem, let alone about good software delivery skills in the context of what the business needs. Anyone you see who appears have all the answers is kidding themselves.
6. In your day job at OpenHFT you deal with hard core low-latency problems. What’s the most difficult set of requirements you’ve had to meet?
As teenager my first computer blew a chip and had to be replaced. The cost of replacing the chip was more than buying a base system second hand, so I bought it from a guy who has a PhD in Psychology. I wondered what he had used it for, as I hadn’t met anyone outside IT who had a computer and he said he was learning to program. When I wonder why he would do that given he had a PhD, he replied; "Computers are easier to fix than people"
The most challenging thing is to workout how to make low latency coding easier for developers to use. With reactive programming and off heap memory, we solve problems in a different way so many inherent issues, which developers have accepted as a given, just don’t occur.
For example, it seems like magic that we can send millions of messages between two processes, persist every message for analysis or replay in testing, and do it without any pauses (or very low pauses) I have worked on Trading systems written in pure Java processes which run all day without a collection. It’s not magic, you create a large Eden space e.g. 24 GB, and you create less than that in a day, so the Eden space doesn’t fill up and no collections occur.
7. OpenHFT has recently relaunched your Chronicle API into several products. Can you give us an explanation of what each of them should be used for?
We had a number of products which were so similar we decided it was simpler to call them all Chronicle. They all persist data off heap, in a GC less way, which can be embedded into multiple processes at the same time.
- Chronicle Engine brings all our products together in an integrated way.
- Chronicle Queue records every event/message in order
- Chronicle Map records the latest data for a key.
- Chronicle Logger provides adapters for popular logging frameworks.
8. Chronicle is well known for its blazing speed, were there any tricks you had to use to get around the JVM’s GC or other limiting factors?
Writing low level Java is about using the portions of Java which are fast. It is a subset of Java. Sometimes, I like writing ‘normal’ Java as it feels like you have so much freedom to write anything which works.
Common patterns of low level Java:
- Create no objects on a per message or per entry basis. Recycle where practical.
- Store the bulk of data off heap.
- Use thread safe off heap structures.
- Use intrinsics instead of JNI. The JVM replaces intrinsics with inlined machine code instructions dramatically reducing their overhead.
9. We’re always interested in performance here at jClarity. Are there any peer reviewed numbers you can share about Chronicle’s performance?
I read vendor’s benchmarks with a degree of scepticism, even my own. A quote referring to how supercomputers are benchmarked; Peak Performance; a manufacturer’s guarantee not to exceed a given performance – Computer Architecture – A Quantitative Approach
As our aim is Simplifying Fast Data, we are more interested in what performance a developer can achieve with minimal effort or input from ourselves. Obvious we want to make money through consulting and support, but I take pride in finding out our product has been running in production trading systems for a year and we didn’t need to know about it.
One of Chronicle’s users at an investment bank reviewed a range of messaging solutions for both interprocess communication on a machine and on a LAN. A number of vendors came in for a couple of days to tune their products. I only found out about it after the review was completed and Chronicle was 8x faster. Now we have far more documentation than we did then. It just worked fast out of the box.
A client I saw last month, said that without Chronicle their tick to trade times was 31 micro-seconds (measured on the wire), and with Chronicle they had got the average down to 23 micro-seconds. Again, without needing our help.
10. Which features in Java do you love to hate the most?
BigDecimal is one of my pet hates, but also one of things I love to hear people using. This is because the use of BigDecimal brings in the most performance consulting work of any Java feature. So while, I love to getting rid of it, please use it and when it performs badly, give me a call.
Another pet hate is StringBuffer. Making it synchronized was always a dumb idea, and yet ten years after a drop in replacement was added, people still use it. I go around removing examples of StringBuffer from stack overflow, like some crazy old guy in the street going through rubbish. If you want to know why it is a dumb idea consider that one method call is thread safe, but two calls are not as there is no locking between calls. Try writing something useful with StringBuffer using only one call e.g. toString() is one call.
11. Which Java code amuses you the most?
InternalFrame InternalFrame Title Pane, Internal Frame Title Pane. Maximize Button Window, Not Focused State.
It could be a strange poem but it’s a class under com.sun.java.swing.plaf.nimbus in Java 6. They shortened in Java 7.
12. What features or changes would you like to see happen in the JVM with regards to getting more throughput?
Getting more throughput should be an easy problem to solve. If you are horizontally scalable you can have more machines. The problem comes when you can’t just add more machines, possibly because the problem you have is not super parallel. This is where low-latency and getting more processing power out of the machines you have is critical. There is little evidence that vendors in the Java space, including Oracle, really understand this. If you charge say $20,000 per CPU, why would you change the code so you needed far less CPUs.
What I would like to see in the JVM are feature which support low latency, esp features which are common in other languages such as stack allocation and data types like BigDecimal being GC-free like double and long are now. I know the JVM has escape analysis, but this rarely works. It’s a hard problem to solve, I am told.
13. What’s the most exciting feature coming up in Java 9/10 for you?
I think the most interesting change in Java 9, is the removal of Unsafe. Unsafe has been a dumping ground for the features which are useful but you are not supposed to do in Java. The Streams and Lambda implementation exceeded my expectations, and I hope the replacement of Unsafe will too.
14. As the CEO of a tech firm, what disruptive trend do you see hitting the Java ecosystem in the next 5 years?
For me, the most disruptive trend is greater use of reactive programming and fast data. Fast data has the potential to reduce latencies to a 1/100th or better and increase throughput by 100x compared to what developers are used to now. This can mean that a whole cloud can be replaced with one machine through more efficient techniques. The main obstacle is that there is not enough developers who understand how to program this way, however this is changing. As machines are getting bigger, you will be able to do more with a single machine. One of my clients has 240 cores and 3 TB of memory, running Java :-).
15. Do you have a blog or twitter account?
OpenHFT’s web site is http://openhft.net/ , all the open source software is Apache 2.0. My blog is http://vanillajava.blogspot.com/ and it has had over 3 million hits. My twitter handle is @PeterLawrey.
Once more we’d like to thank Peter for his detailed and thoughtful answers and encourage all of you to experiment with Chronicle to increase your application’s performance at the sharp end!
Martijn (CEO) and the jClarity Team!