by Bozho | Dec 26, 2020 | Aggregated, agile, Developer tips, Opinions, performance, release
Releasing often is a good thing. It’s cool, and helps us deliver new functionality quickly, but I want to share one positive side-effect – it helps with analyzing production performance issues. We do releases every 5 to 10 days and after a recent release, the application CPU chart jumped twice (the lines are differently colored because we use blue-green deployment): What are the typical ways to find performance issues with production loads? Connect a profiler directly to production – tricky, as it requires managing network permissions and might introduce unwanted overhead Run performance tests against a staging or local environment and do profiling there – good, except your performance tests might not hit exactly the functionality that causes the problem (this is what happens in our case, as it was some particular types of API calls that caused it, which weren’t present in our performance tests). Also, performance tests can be tricky Do a thread dump (and heap dump) and analyze them locally – a good step, but requires some luck and a lot of experience analyzing dumps, even if equipped with the right tools Check your git history / release notes for what change might have caused it – this is what helped us resolve the issue. And it was possible because there were only 10 days of commits between the releases. We could go through all of the commits and spot potential performance issues. Most of them turned out not to be a problem, and one seemingly unproblematic pieces was discovered to be the problem after commenting it out for a brief period a deploying a quick...
by Bozho | Aug 17, 2018 | Aggregated, Developer tips, jackson, JSON, performance
Sometimes you need to export a lot of data to JSON to a file. Maybe it’s “export all data to JSON”, or the GDPR “Right to portability”, where you effectively need to do the same. And as with any big dataset, you can’t just fit it all in memory and write it to a file. It takes a while, it reads a lot of entries from the database and you need to be careful not to make such exports overload the entire system, or run out of memory. Luckily, it’s fairly straightforward to do that, with a the help Jackson’s SequenceWriter and optionally of piped streams. Here’s how it would look like: private ObjectMapper jsonMapper = new ObjectMapper(); private ExecutorService executorService = Executors.newFixedThreadPool(5); @Async public ListenableFuture<Boolean> export(UUID customerId) { try (PipedInputStream in = new PipedInputStream(); PipedOutputStream pipedOut = new PipedOutputStream(in); GZIPOutputStream out = new GZIPOutputStream(pipedOut)) { Stopwatch stopwatch = Stopwatch.createStarted(); ObjectWriter writer = jsonMapper.writer().withDefaultPrettyPrinter(); try(SequenceWriter sequenceWriter = writer.writeValues(out)) { sequenceWriter.init(true); Future<?> storageFuture = executorService.submit(() -> storageProvider.storeFile(getFilePath(customerId), in)); int batchCounter = 0; while (true) { List<Record> batch = readDatabaseBatch(batchCounter++); for (Record record : batch) { sequenceWriter.write(entry); } } // wait for storing to complete storageFuture.get(); } logger.info("Exporting took {} seconds", stopwatch.stop().elapsed(TimeUnit.SECONDS)); return AsyncResult.forValue(true); } catch (Exception ex) { logger.error("Failed to export data", ex); return AsyncResult.forValue(false); } } The code does a few things: Uses a SequenceWriter to continuously write records. It is initialized with an OutputStream, to which everything is written. This could be a simple FileOutputStream, or a piped stream as discussed below. Note that the naming here is a bit misleading – writeValues(out) sounds like you are instructing...
Recent Comments