Integration With Zapier

Integration is boring. And also inevitable. But I won’t be writing about enterprise integration patterns. Instead, I’ll explain how to create an app for integration with Zapier. What is Zapier? It is a service that allows you tо connect two (or more) otherwise unconnected services via their APIs (or protocols). You can do stuff like “Create a Trello task from an Evernote note”, “publish new RSS items to Facebook”, “append new emails to a spreadsheet”, “post approaching calendar meeting to Slack”, “Save big email attachments to Dropbox”, “tweet all instagrams above a certain likes threshold”, and so on. In fact, it looks to cover mostly the same usecases as another famous service that I really like – IFTTT (if this then that), with my favourite use-case “Get a notification when the international space station passes over your house”. And all of those interactions can be configured via a UI. Now that’s good for end users but what does it have to do with software development and integration? Zapier (unlike IFTTT, unfortunately), allows custom 3rd party services to be included. So if you have a service of your own, you can create an “app” and allow users to integrate your service with all the other 3rd party services. IFTTT offers a way to invoke web endpoints (including RESTful services), but it doesn’t allow setting headers, so that makes it quite limited for actual APIs. In this post I’ll briefly explain how to write a custom Zapier app and then will discuss where services like Zapier stand from an architecture perspective. The thing that I needed it for – to be...

GDPR for Developers [presentation]

On a recent meetup in Amsterdam I talked about GDPR from a technical point of view, effectively turning my “GDPR – a practical guide for developers” article into a talk. You can see the slides here: If you’re interested, you can also join a webinar on the same topic, organized by AxonIQ, where I will join Frans Vanbuul. You can find more information about the webinar here. The interesting thing that I can share after the meetup and after meeting with potential clients is that everyone (maybe unsurprisingly) has a very specific question that doesn’t get an immediate answer even after you follow the general guidelines. That is maybe a problem on the Regulation’s side, as it has not brought sufficient clarity to businesses. As I said during the presentation – in technology we’re used with binary questions. In law and legal compliance an answer is somewhere on a scale between 1 and 10. “Do I have to encrypt my data at rest”? Well, I guess yes, but in terms of compliance I’d say “6 out of 10”, as it is not strict, depends on the multiple people’s interpretation of the sensitivity of the data and on other factors like access control. So the communication between legal and technical people is key to understand what exactly implementation changes are needed. The post GDPR for Developers [presentation] appeared first on Bozho's tech...

Fix Your Crawler

Every now and then I open the admin panel of my blog hosting and ban a few IPs (after I’ve tried messaging their abuse email, if I find one). It is always IPs that are generating tons of requests (and traffic) – most likely running some home-made crawler. In some cases the IPs belong to an actual service that captures and provides content, in other cases it’s just a scraper for unknown reasons. I don’t want to ban IPs, especially because that same IP may be reassigned to a legitimate user (or network) in the future. But they are increasing my hosting usage, which in turn leads to the hosting provider suggesting an upgrade in the plan. And this is not about me, I’m just an example – tons of requests to millions of sites are … useless. My advice (and plea) is this – please fix your crawlers. Or scrapers. Or whatever you prefer to call that thing that programmatically goes on websites and gets their content. How? First, reuse an existing crawler. No need to make something new (unless there’s a very specific use-case). A good intro and comparison can be seen here. Second, make your crawler “polite” (the “politeness” property in the article above). Here’s a good overview on how to be polite, including respect for robots.txt. Existing implementations most likely have politeness options, but you may have to configure them. Here I’d suggest another option – set a dynamic crawl rate per website that depends on how often the content is updated. My blog updates 3 times a month – no need to crawl it...

OWASP Dependency Check Maven Plugin – a Must-Have

I have to admit with a high degree of shame that I didn’t know about the OWASP dependency check maven plugin. And seems to have been around since 2013. And apparently a thousand projects on GitHub are using it already. In the past I’ve gone manually through dependencies to check them against vulnerability databases, or in many cases I was just blissfully ignorant about any vulnerabilities that my dependencies had. The purpose of this post is just that – to recommend the OWASP dependency check maven plugin as a must-have in practically every maven project. (There are dependency-check tools for other build systems as well). When you add the plugin it generates a report. Initially you can go and manually upgrade the problematic dependencies (I upgraded two of those in my current project), or suppress the false positives (e.g. the cassandra library is marked as vulnerable, whereas the actual vulnerability is that Cassandra binds an unauthenticated RMI endpoint, which I’ve addressed via my stack setup, so the library isn’t an issue). Then you can configure a threshold for vulnerabilities and fail the build if new ones appear – either by you adding a vulnerable dependency, or in case a vulnerability is discovered in an existing dependency. All of that is shown in the examples page and is pretty straightforward. I’d suggest adding the plugin immediately, it’s a must-have: <plugin> <groupId>org.owasp</groupId> <artifactId>dependency-check-maven</artifactId> <version>3.0.2</version> <executions> <execution> <goals> <goal>check</goal> </goals> </execution> </executions> </plugin> Now, checking dependencies for vulnerabilities is just one small aspect of having your software secure and it shouldn’t give you a false sense of security (a sort-of “I have...

Using Trusted Timestamping With Java

Trusted timestamping is the process of having a trusted third party (“Time stamping authority”, TSA) certify the time of a given event in electronic form. The EU regulation eIDAS gives these timestamps legal strength – i.e. nobody can dispute the time or the content of the event if it was timestamped. It is applicable to multiple scenarios, including timestamping audit logs. (Note: timestamping is not sufficient for a good audit trail as it does not prevent a malicious actor from deleting the event altogether) There are a number of standards for trusted timestamping, the core one being RFC 3161. As most RFCs it is hard to read. Fortunately for Java users, BouncyCastle implements the standard. Unfortunately, as with most security APIs, working with it is hard, even abysmal. I had to implement it, so I’ll share the code needed to timestamp data. The whole gist can be found here, but I’ll try to explain the main flow. Obviously, there is a lot of code that’s there to simply follow the standard. The BouncyCastle classes are a maze that’s hard to navigate. The main method is obviously timestamp(hash, tsaURL, username, password): public TimestampResponseDto timestamp(byte[] hash, String tsaUrl, String tsaUsername, String tsaPassword) throws IOException { MessageImprint imprint = new MessageImprint(sha512oid, hash); TimeStampReq request = new TimeStampReq(imprint, null, new ASN1Integer(random.nextLong()), ASN1Boolean.TRUE, null); byte[] body = request.getEncoded(); try { byte[] responseBytes = getTSAResponse(body, tsaUrl, tsaUsername, tsaPassword); ASN1StreamParser asn1Sp = new ASN1StreamParser(responseBytes); TimeStampResp tspResp = TimeStampResp.getInstance(asn1Sp.readObject()); TimeStampResponse tsr = new TimeStampResponse(tspResp); checkForErrors(tsaUrl, tsr); // validate communication level attributes (RFC 3161 PKIStatus) tsr.validate(new TimeStampRequest(request)); TimeStampToken token = tsr.getTimeStampToken(); TimestampResponseDto response = new TimestampResponseDto(); response.setTime(getSigningTime(token.getSignedAttributes())); response.setEncodedToken(Base64.getEncoder().encodeToString(token.getEncoded()));...

GDPR – A Practical Guide For Developers

You’ve probably heard about GDPR. The new European data protection regulation that applies practically to everyone. Especially if you are working in a big company, it’s most likely that there’s already a process for gettign your systems in compliance with the regulation. The regulation is basically a law that must be followed in all European countries (but also applies to non-EU companies that have users in the EU). In this particular case, it applies to companies that are not registered in Europe, but are having European customers. So that’s most companies. I will not go into yet another “12 facts about GDPR” or “7 myths about GDPR” posts/whitepapers, as they are often aimed at managers or legal people. Instead, I’ll focus on what GDPR means for developers. Why am I qualified to do that? A few reasons – I was advisor to the deputy prime minister of a EU country, and because of that I’ve been both exposed and myself wrote some legislation. I’m familiar with the “legalese” and how the regulatory framework operates in general. I’m also a privacy advocate and I’ve been writing about GDPR-related stuff in the past, i.e. “before it was cool” (protecting sensitive data, the right to be forgotten). And finally, I’m currently working on a project that (among other things) aims to help with covering some GDPR aspects. I’ll try to be a bit more comprehensive this time and cover as many aspects of the regulation that concern developers as I can. And while developers will mostly be concerned about how the systems they are working on have to change, it’s not unlikely...