by Bozho | Dec 2, 2016 | Aggregated, Opinions
I’ve been an advisor to the depury prime minister of Bulgaria for the past year and a half. And on this year’s OpenFest conference I tried to report on what we’ve achieved. It is not that much and there are no visible results, which is a bit disappointing, but we (a small motivated team) believe we have laid the groundwork for a more open, and properly built ecosystem for the government IT systems. Just to list a few things – we passed a law that requires open sourcing custom-built government software, we opened a lot of data (1500 datasets) on the national open data portal, and we drew a roadmap of how existing state registers and databases be upgraded in order to meet modern software engineering best practices and be ready to meet the high load of requests. We also seriously considered the privacy and auditability of the whole ecosystem. We prepared the electronic identification project (each citizen having the option to identify online with a secure token), an e-voting pilot and so on. The video of the talk is available here: And here are the slides: Now that our term is at an end (due to the resignation of the government) we hope the openness-by-default will persist as a policy and the new government agency that we constituted would be able to push the agenda that has been laid out. Whether that will be the case in a complex political situation is hard to tell, but hopefully the “technical” and the “political” aspects won’t be entwined in a negative way. And our team will continue to support (even...
by Bozho | Nov 19, 2016 | Aggregated, Developer tips
As a consequence of the Dyn attack many major websites were down, including twitter – the browsers could not resolve an IP address of the servers because the authoritative name server (Dyn) was down. Whether that could be addressed globally, I don’t know – there was an interesting discussion on reddit about my proposal to increase TTL – how the resolution policy and algorithms can be improved, why a lower TTL is not always applicable, etc. But while twitter.com was down, the mobile app was also not working. And while we have no control over the browser, we certainly do have control on the mobile app (same goes for desktop applications, of course, but I’ll be talking mainly about mobile apps as more dominant). The reason the app was also down is that is most likely uses the twitter.com domain as well (e.g. api.twitter.com). And that’s the right way to do it, except in these rare situations when the DNS fails. In these cases you can hardcode a list of server IP addresses in your app and fallback to them if the domain-name based requests fail (e.g. after 3 attempts). If you change your server IP addresses, you just update the app. It doesn’t matter that it will have an unpredictable delay (until everyone updates) and that some clients won’t have a proper IP – it is an edge-case fallback mechanism. It’s not of course my idea – it’s been used by distributed systems like Bitcoin and BitTorrent – for example, when trying to join the network, the Bitcoin or a BitTorrent DHT client tries to connect to a...
by Bozho | Nov 8, 2016 | Aggregated, Developer tips
In a beginner’s guide to concurrency, I mentioned advisory locks. These are not the usual table locks – they are table-agnostic, database-specific way to obtain a named lock from your application. Basically, you use your database instance for centralized application-level locking. What could it be used for? If you want to have serial operations, this is a rather simple way – no need for message queues, or distributed locking libraries in your application layer. Just have your application request the lock from the database, and no other request (regardless of the application node, in case there are multiple) can obtain the same lock. There are multiple functions that you can use to obtain such a lock – in PostgreSQL, in MySQL. The implementations differ slightly – in MySQL you need to explicitly release the lock, in PostgreSQL a lock can be released at the end of the current transaction. How to use it in a Java application, for example with spring. You can provide a locking aspect and a custom annotation to trigger the locking. Let’s say we want to have sequential updates for a given entity. In the general use-case that would be odd, but sometimes we may want to perform some application-specific logic that relies on sequential updates. @Before("execution(* *.*(..)) && @annotation(updateLock)") public void applyUpdateLocking(JoinPoint joinPoint, UpdateLock updateLock) { int entityTypeId = entityTypeIds.get(updateLock.entity()); // note: letting the long id overflow when fitting into an int, because the postgres lock function takes only ints // lock collisions are pretty unlikely and their effect will be unnoticeable int entityId = (int) getEntityId(joinPoint.getStaticPart().getSignature(), joinPoint.getArgs(), updateLock.idParameter()); if (entityId != 0) {...
by Bozho | Oct 22, 2016 | Aggregated, Developer tips, Opinions
Yesterday Dyn, a DNS-provider, went down after a massive DDoS. That led to many popular websites being inaccessible, including twitter, LinkedIn, eBay and others. The internet seemed to be “crawling on its knees”. We’ll probably read an interesting post-mortem from Dyn, but why did that happen? First, DDoS capacity is increasing, using insecure and infected IoT devices with access to the internet. Huge volumes of fake requests are poured on a given server or set of servers and they become inaccessible, either being unable to cope with the requests, or simply because the network to the server doesn’t have enough throughput to accomodate all the requests. But why did “the internet” stop because a single DNS provider was under attack? First, because of centralization. The internet is supposed to be decentralized (although I’ve argued that exactly because of DNS, it is pseudo-decentralized). But services like Dyn, UltraDNS, Amazon Route53 and also Akamai and CloudFlare centralize DNS. I can’t tell how exactly, but out of the top 500 websites according to Moz.com, 181 use one of the above 5 services as their DNS provider. Add 25 google services that use their own, and you get nearly 200 out of 500 centered in just 6 entities. But centralization of the authoritative nameservers alone would not have led to yesterday’s problem alone. A big part of the problem, I think, is the TTL (time to live) of the DNS records, that is – the records which contain the mapping between domain name and IP address(es). The idea is that you should not always hit the authoritative nameserver (Dyn’s server(s) in this case)...
by Bozho | Oct 12, 2016 | Aggregated, Opinions
I’ll being this post with a rant about the state of scientific publishing, then review the technology “disruption” landscape and offer a partial improvement that I developed (source). Scientific publishing is quite important – all of science is based on previously confirmed “science”, so knowing what the rest of the scientific community has done or is doing is essential to research. And allows scientists to “stand on the shoulders of giants”. The web was basically invented to improve the sharing of scientific information – it was created at CERN and allowed linking from one (research) document to others. However, scientific publishing at the moment is one of the few industries that haven’t benefited from the web. Well, the industry has – the community hasn’t, at least not as much as one would like. Elsevier, Thomson-Reuters, Springer and other publishers make huge profits (e.g. 39% margin on a 2 billion revenue) for doing something that should basically be free in this century – they spread the knowledge that scientists have created. You can see here some facts about their operation, the most striking being that each university has to pay more than a million dollars to get the literature it needs. It’s because they rely on a centuries old process of submission to journals, accepting the submission, then printing and distributing to university libraries. Recently publishers have put publications online, but they are behind paywalls or accessible only after huge subscription fees have been paid. I’m not a “raging socialist” but sadly, publishers don’t provide (sufficient) value. They simply gather the work of scientists that is already funded by public...
Recent Comments