by Bozho | Nov 10, 2018 | Aggregated, Developer tips, dht, distributed, Opinions, p2p
Distributed p2p technologies have always been fascinating to me. Bittorrent is cool not because you can download pirated content for free, but because it’s an amazing piece of technology. At some point I read and researched a lot about how DHTs (distributed hash tables) work. DHTs are not part of the original bittorrent protocol, but after trackers were increasingly under threat to be closed for copyright infringment, “trackerless” features were added to the protocol. A DHT is distributed among all peers and holds information about which peer holds what data. Once you are connected to a peer, you can query it for their knowledge on who has what. During my research (which was with no particular purpose) I took a note on many resources that I thought useful for understanding how DHTs work and possibly implementing something ontop of them in the future. In fact, a DHT is a “shared database”, “just like” a blockchain. You can’t trust it as much, but proving digital events does not require a blockchain anyway. My point here is – there is a lot more cool stuff to distributed / p2p systems than blockchain. And maybe way more practical stuff. It’s important to note that the DHT used in BitTorrent is Kademlia. You’ll see a lot about it below. Anyway, the point of this post is to share the resources that I collected. For my own reference and for everyone who wants to start somewhere on the topic of DHTs. Bittorrent DHT protocol – a nice explanation how DHT is used in bittorent (here’s a list of all bittorrent protocol enhancements) Kademlia: design...
by Bozho | Nov 3, 2018 | Aggregated, ethics, machine learning, Opinions
Today I had a talk on OpenFest about algorithmic and technological transparency. It was a somewhat abstract and high-level talk but I hoped to make it more meaningful than if some random guy just spoke about techno-dystopias. I don’t really like these kinds of talks where someone who has no idea what a “training set” is, how dirty the data is and how to write five lines in Python talks about an AI revolution. But I think being a technical person let’s me put some details into the high-level stuff and make it less useless. The problem I’m trying to address is opaque systems – we don’t know how recommendation systems work, why are seeing certain videos or ads, how decisions are taken, what happens with our data and how secure the systems we use are, including our cars and future self-driving cars. Algorithms have real impact on society – recommendation engines make conspiracy theories mainstream, motivate fascists, create echo chambers. Algorithms decide whether we get loans, welfare, a conviction, or even if we get hit by a car (as in the classical trolley problem). How do we make the systems more transparent, though? There are many approaches, some overlapping. Textual description of how things operate, tracking the actions of back-office admins and making them provable to third parties, implementing “Why am I seeing this?”, tracking each step in machine learning algorithms, publishing datasets, publishing source code (as I’ve previously discussed open-sourcing some aspects of self-driving cars). We don’t have to reveal trade secrets and lose competitive advantage in order to be transparent. Transparency doesn’t have to come at...
by Bozho | Oct 22, 2018 | Aggregated, Opinions
Every now and then you start thinking “wouldn’t it be nice to have an app for X”. When I was in that situation, I took a note. Then that note grew, I cleaned up some absurd ones and added some more. I implemented some of these ideas of mine, the rest formed a “TODO” list. At some point I realized I won’t be able to implement them, as the time needed is not something I’m likely to have in the near future. But I didn’t like to just delete the notes. So here they are – my random app ideas. Probably useless, but maybe a little interesting. Receipts via smartphone NFC – there are apps that let you track your expenses, but they are tedious. There are apps that try to OCR receipts, but they vary too much in different parts of the world for that to be consistent (there are companies like Receipt bank that do something like that, but it’s not the way I’d like this problem solved in the long run). So I thought stores can offer the option for NFC receipts – you just tap your phone to a device (or another phone) and get the receipt in an electronic form. Not the picture, but the raw data – items, prices, taxes, issuer. Then you have to option to print it if you like. Of course I realized at some point that first legislation should allow for that – in many cases you must issue a paper receipt and the digital one is not “legal”. But the idea still remains viable and probably not hard...
by Bozho | Sep 29, 2018 | Aggregated, electronic identification, Opinions, security, smartcards
Electronic identity is an important concept as it lies on the crossroads of the digital, the physical and the legal worlds. How do you prove your identity without being physically present? I’ve previously given an overview talk on electronic identification, tackled the high-level and philosophical aspects, and criticized biometric-only identification as insecure. Now I want to list a few practical ways in which eID can be implemented. First, it’s important to mention the eIDAS regulation which makes electronic identity a legal reality. It says what an electronic identification scheme is and what are its legal effects (proving that you are who you claim to be). And it defines multiple “levels of assurance”, i.e. the levels of security of a method of authentication. But it is a broad framework that doesn’t tell you how to do things in the technical world. And while electronic identification is mostly cited in the context of government and municipal services, it applies to private companies as well. Currently in the US, for example, the SSN is used for electronic identification. This is a very poor approach, as leaking the SSN allows for identity theft. In the EU there are many different approaches to do that, from the Estonian PKI-based approach, to UK’s verify initiative, which relies on the databases of private companies. You can see electronic identity as a more legally-meaningful login. You still perform a login, in many cases using username and password as one of the factors, but it carries additional information – who is the actual person behind that login. In some cases it doesn’t have to even give information on...
by Bozho | Sep 24, 2018 | Aggregated, Developer tips, Opinions
You may think you have logs. Chances are, you can rely on them only for tracing exceptions and debugging. But you can’t rely on them for compliance, forensics, or any legal matter. And that may be none of your concern as an engineer, but it is one of those important non-functional requirements that, if not met, are ultimately our fault. Because of my audit trail startup, I’m obviously both biased and qualified to discuss logs (I’ve previously described what audit trail / audit logs are and how they can be maintained). And while they are a only a part of the security aspects, they are important, especially from a legal standpoint. That’s why many standards and laws – including ISO 27001, PCI-DSS, HIPAA, SOC 2, PSD2, GDPR – require audit trail functionality. And most of them have very specific security requirements. PCI-DSS has a bunch of sections with audit trail related requirements: 10.2.3 “Malicious users often attempt to alter audit logs to hide their actions, and a record of access allows an organization to trace any inconsistencies or potential tampering of the logs to an individual account. [..]” 10.5 Secure audit trails so they cannot be altered. Often a malicious individual who has entered the network will attempt to edit the audit logs in order to hide their activity. Without adequate protection of audit logs, their completeness, accuracy, and integrity cannot be guaranteed, and the audit logs can be rendered useless as an investigation tool after a compromise. 10.5.5 Use file-integrity monitoring or change-detection software on logs to ensure that existing log data cannot be changed without generating alerts...
by Bozho | Jul 29, 2018 | Aggregated, blockchain, digital signatures, Opinions, security
Recently technical and non-technical people alike started to believe that the best (and only) way to prove that something has happened in an information system is to use a blockchain. But there are other ways to achieve that that are arguably better and cheaper. Of course, blockchain can be used to do that, and it will do it well, but it is far from the only solution to this problem. The way blockchain proves that some event has occurred by putting it into a tamper-evident data structure (a hash chain of the roots of merkle trees of transactions) and distributing that data structure across multiple independent actors so that “tamper-evident” becomes “tamper-proof” (sort-of). So if an event is stored on a blockchain, and the chain is intact (and others have confirmed it’s intact), this is a technical guarantee that it had indeed happened and was neither back-dated, nor modified. An important note here – I’m stressing on “digital” events, because no physical event can be truly guaranteed electronically. The fact that someone has to enter the physical event into a digital system makes this process error-prone and the question becomes “was the event correctly recorded” rather than “was it modified once it was recorded”. And yes, you can have “certified” / “approved” recording devices that automate transferring physical events to the digital realm, e.g. certified speed cameras, but the certification process is a separate topic. So we’ll stay purely in the digital realm (and ignore all provenance use cases). There are two aspects to proving digital events – technical and legal. Once you get in court, it’s unlikely to...
Recent Comments