by Bozho | Jun 15, 2020 | Aggregated, cache, Developer tips, spring
In a third post about cache managers in spring (over a long period of time), I’d like to expand on the previous two by showing how to configure multiple cache managers that dynamically create caches. Spring has CompositeCacheManager which, in theory, should allow using more than one cache manager. It works by asking the underlying cache managers whether they have a cache with the requested name or not. The problem with that is when you need dynamically-created caches, based on some global configuration. And that’s the common scenario, when you don’t want to manually define caches, but instead want to just add @Cacheable and have spring (and the underlying cache manager) create the cache for you with some reasonable defaults. That’s great until you need to have more than one cache managers. For example – one for local cache and one for a distributed cache. In many cases a distributed cache is needed; however not all method calls need to be distributed – some can be local to the instance that handles it and you don’t want to burden your distributed cache with stuff that can be kept locally. Whether you can configure a distributed cache provider to designate some cache to be local, even though it’s handled by the distributed cache provider – maybe, but I don’t guarantee it will be trivial. So, faced with that issue, I had to devise some simple mechanism of designating some caches as “distributed” and some as “local”. Using CompositeCacheManager alone would not do it, so I extended the distributed cache manager (in this case, Hazelcast, but it can be done with...
by Bozho | May 5, 2020 | Aggregated, Developer tips
Mr. Robot is an amazingly accurate series about a hacker (Elliot Alderson) and his, uhm, undertakings. The series is consulted by many cybersecurity experts and so every hack that happens is actually properly executed, using real tools and commands that do exactly what an infosec expert would expect. Nothing shown on screen is the usual bullshit TV hacking. And that is awesome and scary in a number of ways. Obviously, this post is full of spoilers, so SPOILER ALERT, if you haven’t watched it, go and do that and then continue reading. Another disclaimer I have to make is that you should use the tools and techniques below only for ethical hacking, penetration testing and other legal activities. The series has many hacks happening, from connecting to a neighbor’s WiFi, guessing people’s social media passwords and installing malware via flash drives, to complex social engineering plots, an HSM hack, the use of steganography and other advanced stuff. There’s even a tool pack with everything that’s used in the series. I’m fairly well-versed in the cybersecurity domain so even though I haven’t performed any hacks, most of the things happening on screen were familiar. But I did learn some interesting things from the series, which I’d like to share, as they sparked my interest and made me read a bit more. Many hacking scenes require you to pause and try to read what’s on screen, but that makes it even more fun. proxychains – many tools are in use in the series, and Kali Linux is often used. But I wasn’t familiar with one particular tool that I think is...
by Bozho | Apr 17, 2020 | Aggregated, alarms, alerts, aws, Developer tips, devops
Monitoring is key for any real-world application. You have to know what’s happening and be alerted in real time if something wrong is happening. AWS has CloudWatch for that, and gives you a lot of metrics automatically. But there are some that you have to define yourself. And then you need to define proper alarms. Here I’ll focus on hour: High number of application errors High number of application warnings High number of 5xx errors on the load balancer High number of 4xx errors on the load balancer First, the prerequisites: You need to be using CloudFormation to automate everything. You can create all of those things manually, but automation is a big plus If using CloudFormation, you’d preferably have a sub-stack for configuring alarms You need to be collecting your logs with CloudWatch logs If you are not using CloudWatch logs, here’s a simple config file and script to enable them: { "agent": { "metrics_collection_interval": 10, "region": "eu-west-1", "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log" }, "logs": { "logs_collected": { "files": { "collect_list": [ { "file_path": "{{logPath}}", "log_group_name": "{{logGroupName}}", "log_stream_name": "{instance_id}", "timestamp_format": "%Y-%m-%d %H:%M:%S" } ] } } } } # install AWS CloudWatch monitor mkdir cloud-watch-agent cd cloud-watch-agent wget https://s3.amazonaws.com/amazoncloudwatch-agent/linux/amd64/latest/AmazonCloudWatchAgent.zip unzip AmazonCloudWatchAgent.zip ./install.sh aws s3 cp s3://$BUCKET_NAME/cloudwatch-agent-config.json /var/config/cloudwatch-agent-config.json sed -i -- 's|{{logPath}}|/var/log/application.log|g' /var/config/cloudwatch-agent-config.json sed -i -- 's|{{logGroupName}}|app_node|g' /var/config/cloudwatch-agent-config.json sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/var/config/cloudwatch-agent-config.json -s Now you have to define two things: Log metrics and alarms. The cloudformation code below creates both: "HighAppErrorsNotification": { "Type": "AWS::CloudWatch::Alarm", "Properties": { "AlarmActions": [ { "Ref": "NotificationTopicId" } ], "InsufficientDataActions": [ { "Ref": "NotificationTopicId" } ], "AlarmDescription": "Notify if there are too many application...
by Bozho | Mar 19, 2020 | Aggregated, Developer tips, wordpress
WordPress is powering 35% of website. And while it may not be seen as very complex or interesting, it is one of the most prevalent technologies of our time. And many developers, even if they are not working with PHP, have to support some WordPress installation (e.g. a blog like this one). And unfortunately, there are still basic things that you can’t easily do. Plugins do help, but they don’t always have the proper functionality. Today I had to change permalinks on a website. From https://example.com/post-name to https://example.com/blog/post-name. And WordPress allows that, except there’s a problem – when you change it, the old links stop working (404). Not only that ruins your SEO, any previous share of your post will not work. I’ve pointed that out a long time ago when Spring broke their documentation links. Bottomline: you don’t want that to happen. Instead, you want to do 301 redirect (and not 302, which appears to also break your SEO). But all tutorials that are easily found online assume you can manually configure redirection through some plugin. But if you have 100-200 posts, that’s a lot of tedious work. There are plugins that allegedly monitor your posts for changes and create the redirects automatically. That may work if you manually edit a post, but it didn’t work for me when changing the permalink format settings. So how to do it in an automated way and without disrupting your website? You’d need a little bit more than a plugin – namely SQL and Regex. Here are the steps: Install and activate the Redirection plugin Open your blog’s hosting admin page...
by Bozho | Mar 13, 2020 | Aggregated, aws, Developer tips, elasticsearch, Opinions, post-mortem
So it happened that we had a production issue on the SaaS version of LogSentinel – our Elasticsearch stopped indexing new data. There was no data loss, as elasticsearch is just a secondary storage, but it caused some issues for our customers (they could not see the real-time data on their dashboards). Below is a post-mortem analysis – what happened, why it happened, how we handled it and how we can prevent it. Let me start with a background of how the system operates – we accept audit trail entries (logs) through a RESTful API (or syslog), and push them to a Kafka topic. Then the Kafka topic is consumed to store the data in the primary storage (Cassandra) and index it for better visualization and analysis in Elasticsearch. The managed AWS Elasticsearch service was chosen because it saves you all the overhead of cluster management, and as a startup we want to minimize our infrastructure management efforts. That’s a blessing and a curse, as we’ll see below. We have alerting enabled on many elements, including the Elasticsearch storage space and the number of application errors in the log files. This allows us to respond quickly to issues. So the “high number of application errors” alarm triggered. Indexing was blocked due to FORBIDDEN/8/index write. We have a system call that enables it, so I tried to run it, but after less than a minute it was blocked again. This meant that our Kafka consumers failed to process the messages, which is fine, as we have a sufficient message retention period in Kafka, so no data can be lost. I...
by Bozho | Feb 22, 2020 | Aggregated, aws, database, Developer tips, scalability
When you have to run a scalable application on AWS, your database must also be scalable. It’s easier to scale the stateless application layer, where each node is mostly disposable – even if a node in a 3-node cluster fails, you can just fire up another one and nobody notices. The database layer is stateful and therefore there’s a risk to lose data. Having just a single node is not an option, as a node can always go down and that would mean downtime. So you need multiple nodes in a cluster to make sure your application is highly available and fault tolerant (I won’t go into the differences in terminology). What database am I talking about? It doesn’t matter. It can be a SQL or a NoSQL database – each has some form of clustering available. Whether it’s active-active or active-passive. Now, for AWS in particular, you can choose RDS (or another managed option), which will handle it for you. But if there’s no managed option (e.g. Cassandra) or you don’t feel the managed option is giving you enough control, or is more expensive, or the version you require is not available, you have to manage the database layer yourself. I won’t go into the details of how to configure the database-specific clustering – you should check the documentation of the particular database for that. I’ll try to give some tips how to safely run your infrastructure that supports the database cluster. And here come auto-scaling groups. They allow you have have a group of identical nodes (based on a launch configuration) and the ASG makes sure you...
Recent Comments