by Bozho | Aug 4, 2014 | Aggregated, Opinions
One of the issues with spam is false positives. “Did you check your spam folder” is often a question to ask if your email is not received on the other end. I’m not a machine learning expert and I’ve never made a spam filter, and I only know the naive Bayes approach. So this suggestion is not a machine-learning “breakthrough”. But from what I know about classification algorithms is that they usually provide a likelihood of one item being in one group or another. Some items are not identified as spam with absolute certainty – they are 51% likely to be spam, for example. My suggestions is: for borderline items (lower certainty that they should be classified as spam), the spam filter should send emails to the sender indicating that his message was considered spam. A genuine sender will probably take additional steps, like sending another short email or calling/messaging the recipient (‘click here to confirm you are not spam’ won’t work, because it will easily be automated). It’s rather a usability suggestion than a technical one, and I’m sure there are some issues that I’m missing. But I thought it’s at least worth sharing. One of the issues with spam is false positives. “Did you check your spam folder” is often a question to ask if your email is not received on the other end. I’m not a machine learning expert and I’ve never made a spam filter, and I only know the naive Bayes approach. So this suggestion is not a machine-learning “breakthrough”. But from what I know about classification algorithms is that they usually provide a...
by Bozho | Jul 14, 2014 | Aggregated, Opinions
The other day I gave a presentation about “The Cloud”. I talked about buzzwords, incompetence, classification, and most importantly – embracing failure. Here are the slides (the talk was not in English). I didn’t have time to go into too much details, but I hope it’s a nice overview. The other day I gave a presentation about “The Cloud”. I talked about buzzwords, incompetence, classification, and most importantly – embracing failure. Here are the slides (the talk was not in English). I didn’t have time to go into too much details, but I hope it’s a nice...
by Bozho | Jul 3, 2014 | Aggregated, Developer tips, Opinions
I’m a minimalist, and I don’t like to complicate software too early and unnecessarily. And adding components to a software system is one of the things that adds a significant amount of complexity. So let’s talk about message queues. Message Queues are systems that let you have fault-tolerant, distributed, decoupled, etc, etc. architecture. That sounds good on paper. Message queues may fit in several use-cases in your application. You can check this nice article about the benefits of MQs of what some use-cases might be. But don’t be hasty in picking an MQ because “decoupling is good”, for example. Let’s use an example – you want your email sending to be decoupled from your order processing. So you post a message to a message queue, then the email processing system picks it up and sends the emails. How would you do that in a monolithic, single classpath application? Just make your order processing service depend on an email service, and call sendEmail(..) rather than sendToMQ(emailMessage). If you use MQ, you define a message format to be recognized by the two systems; if you don’t use an MQ you define a method signature. What is the practical difference? Not much, if any. But then you probably want to be able to add another consumer that does additional thing with a given message? And that might happen indeed, it’s just not for the regular project out there. And even if it is, it’s not worth it, compared to adding just another method call. Coupled – yes. But not inconveniently coupled. What if you want to handle spikes? Message queues give you...
by Bozho | Jun 25, 2014 | Aggregated, Opinions
We’ve all had incompetent colleagues. People that tend to write bad code, make bad decisions or just can’t understand some of the concepts in the project(s). And it’s never trivial to handle this scenario. Obviously, the easiest solution is to ignore it. And if you are not a team lead (or something similar), you can probably pretend that the problem doesn’t exist (and occasionally curse and refactor some crappy code). There are two types of incompetent people: those who know they are not that good, and those who are clueless about their incompetence. The former are usually junior and mid-level developers, and they are expected to be less experienced. With enough coaching and kindly pointing out their mistakes, they will learn. This is where all of us have gone though. The latter is the harder breed. They are the “senior” developers that have become senior only due to the amount of years they’ve spent in the industry, and regardless of their actual skills or contribution. They tend to produce crappy code, misunderstand assignments, but on the other hand reject (kindly or more aggressively) any attempt to be educated. Because they’re “senior”, and who are you to argue with them? In extreme cases this may be accompanied with an inferiority complex, which in turn may result in clumsy attempts to prove they are actually worthy. In other cases it may involve pointless discussions on topics they do not want to admit they are wrong about, just because admitting that would mean they are inferior. They will often use truisms and general statements instead of real arguments, in order to show...
by Bozho | May 27, 2014 | Aggregated, Opinions
E-voting is a hot topic in my country, and has been discussed a lot everywhere. Since we are already using the internet and touch-screen technologies in our everyday lives, why not apply that to voting? And not for the sake of technology itself, but in order to prevent technical mistakes and election fraud, and make it easier for citizens to cast their vote and make the elections generally cheaper. There are many concerns, some of which – relevant, including security, single points of failure, privacy, etc. Some experts claim it is impossible to make it secure enough, and that paper ballots must be used forever. On the other hand, there are several companies producing voting machines, and multiple attempts have been made to introduce e-voting, very few of which were successful. A recent audit of the Estonian e-voting system also showed some drawbacks, although the system has been in use for a while without major issues. I’ve been thinking and discussing about the details of how a system for electronic voting can be implemented, with the following main requirements: the results cannot be tampered with – neither by an attacker, nor by the election authorities open source – relying on closed source and private audits is “security through obscurity” everyone can vote – there should be no technical limitation to voting – people without internet and without profound technology skills should be able to cast a vote guaranteed anonymity – nobody should be able to see how a person voted only one vote per person – the system must be able to ensure that a person hasn’t voted more...
by Bozho | May 19, 2014 | Aggregated, Developer tips, Opinions
Every now and then I make some addition to my algorithmic music composer. And now I decided to allow users to hear music that is based on their tweets. It is nothing extraordinary, but I wanted to get some experience with basic natural language processing, so I decided to analyze the latest 200 tweets of the user, and perform the following analysis, which in turn influences the music that the user gets: Sentiment analysis – I pass each tweet to the Stanford deep learning sentiment analysis (part of CoreNLP), which influences the musical scale – if the average sentiment of all tweets is: positive => major scale, negative => minor scale. Neutral leads to either lydian or dorian scale. Let me elaborate on “average sentiment” – the sentiment of each tweet is a number between 0 (very negative) and 4 (very positive). The average then, obviously, is the sum of the sentiments / number tweets. Tweet length – if your tweets are on average shorter than 40 characters, the scale is pentatonic. Otherwise it’s heptatonic Tweet frequency – the average interval between your tweets determines the tempo of the music. The more tweets you post (and the lower the time between them is), the faster the music. Your tweet tempo is your tweets’ music tempo. Variation – now that’s the most fuzzy metric, and the algorithm that I use is the most naive one. I extract all words from all tweets, apply a stemmer (that is, transform them to their base form, e.g. eats -> eat, thieves -> thief, etc., again part of CoreNLP), remove the stop words (and,...
Recent Comments