by Bozho | May 22, 2014 | Aggregated, Developer tips
Our systems often depend on 3rd party services (They may even be services internal to the company that we have no control on). Such services include Social Networks exposing APIs, SaaS with APIs like Salesforce, Authentication providers, or any system that our system communicates with, but is outside our product lifecycle. In regular integration tests, we would have an integration deployment of all sub-systems, in order to test how they work together. In case of external services, however, we can only work with the real deployment (given some API credentials). What options do we have to write integration tests, i.e. check if our system properly integrates with the external system? If the service provides a sandbox, that’s the way to go – you have a target environment where you can do anything and it will be short-lived and not visible to any end-users. This is, however, rare, as most external services do not provide such sandboxes. Another option is to have an integration test account – e.g. you register an application at twitter, called “yourproduct-test”, create a test twitter account, and provide these credentials to the integration test. That works well if you don’t have complex scenarios involving multi-step interactions and a lot of preconditions. For example, if you application is analyzing tweets over a period of time, you can’t post tweets with the test account in the past. The third option is mocks. Normally, mocks and integration tests are mutually exclusive, but not in this case. You don’t want to test whether the external service conforms to its specification (or API documentation) – you want to test whether...
by Bozho | May 19, 2014 | Aggregated, Developer tips, Opinions
Every now and then I make some addition to my algorithmic music composer. And now I decided to allow users to hear music that is based on their tweets. It is nothing extraordinary, but I wanted to get some experience with basic natural language processing, so I decided to analyze the latest 200 tweets of the user, and perform the following analysis, which in turn influences the music that the user gets: Sentiment analysis – I pass each tweet to the Stanford deep learning sentiment analysis (part of CoreNLP), which influences the musical scale – if the average sentiment of all tweets is: positive => major scale, negative => minor scale. Neutral leads to either lydian or dorian scale. Let me elaborate on “average sentiment” – the sentiment of each tweet is a number between 0 (very negative) and 4 (very positive). The average then, obviously, is the sum of the sentiments / number tweets. Tweet length – if your tweets are on average shorter than 40 characters, the scale is pentatonic. Otherwise it’s heptatonic Tweet frequency – the average interval between your tweets determines the tempo of the music. The more tweets you post (and the lower the time between them is), the faster the music. Your tweet tempo is your tweets’ music tempo. Variation – now that’s the most fuzzy metric, and the algorithm that I use is the most naive one. I extract all words from all tweets, apply a stemmer (that is, transform them to their base form, e.g. eats -> eat, thieves -> thief, etc., again part of CoreNLP), remove the stop words (and,...
by Bozho | May 11, 2014 | Aggregated, Opinions
Recently I’ve been trying to get a bit into music theory, machine learning, computational linguistics, so I ended up looking at libraries and tools written by the scientific community – examples include the Stanford Core NLP library, GATE, Weka, jMusic, and several more. The general feeling is that scientific libraries have mostly bad code. I will not point fingers, but there are too many freshman mistakes – not considering thread-safety, cryptic, ugly and/or stringly-typed APIs, lack of type-safety, poorly named variables and methods, choosing bad/slow serialization formats, writing debug messages to System.err (or out), lack of documentation, lack of tests. Thus using these libraries becomes time consuming and error prone. Every 10 minutes you see some horribly written code that you don’t have the time to fix. And it’s not just one or two things, that you would report in a normal open-source project – it’s an overall low level of quality. On the other hand these libraries have a lot of value, because the low-level algorithms will take even more time and especially know-how to implement, so just reusing them is obviously the right approach. Some libraries are even original research and so you just can’t write them yourself, without spending 3 years on a PhD thesis. I cannot but mention Heartbleed here – OpenSSL is written by scientific people, and much has been written on topic that even OpenSSL does not meet modern software engineering standards. But that’s only the surface. Scientists in general can’t write good code. They write code simply to achieve their immediate goal, and then either throw it away, or keep using it...
by Bozho | Apr 25, 2014 | Aggregated, Opinions
Below are several development “methodologies” that are popular and even industry-standard: Hype-Driven Development – you are either a startup, or you are given the freedom to choose whatever technology you like for your new cool-cutting-edge-distrupting-innovative-did-I-say-cool project. What technologies to use? The recently overhyped ones, of course. Let’s do it in Node.js, and you have to make it reactive, and do DevOps, even though your Linux experience is limited to running Ubuntu on your desktop, and store the data in MongoDB (it’s web-scale!). Speaking of web-scale, you will obviously need to go to the cloud. Then all of a sudden you have an unintelligible codebase, broken servers, pages take a second to load, you lose data every now and then, and you are unable to actually scale. But hey, you used cutting-edge, web-scale reactive technologies that you didn’t understand when you started, and were not applicable to your domain. But now your write a blogpost describing how to solve a problem with them, that has been solved for decades with other technologies. And you post it to hacker news, contributing to the hype. Demo-Driven Development – you work in a team, part of a big organization that has adopted Agile/SCRUM. Or you are a “lean startup” and define the projects on the fly. In both cases the project does not have a clearly defined goal, but somehow money are being poured into it, so it has to keep going. The end result doesn’t seem to matter, but process has to be followed and you need to be be able to demo stuff to the stakeholders. So you write your code...
by Bozho | Apr 18, 2014 | Aggregated, Developer tips
A somewhat common usecase for android applications is to have them launched when the phone is started, and execute some piece of code periodically. Sounds straightforward, but there are some pitfalls, so here are a couple of steps to take in order to achieve that. Start with the trivial stuff: In your manifest you need the following in order to be able to receive the boot event: <uses-permission android:name="android.permission.RECEIVE_BOOT_COMPLETED" /> Then, again in the manifest, you need two receivers: <receiver android:name=".StartupLauncher"> <intent-filter> <action android:name="android.intent.action.BOOT_COMPLETED" /> <action android:name="android.intent.action.QUICKBOOT_POWERON" /> <category android:name="android.intent.category.DEFAULT" /> </intent-filter> </receiver> <receiver android:name=".Notifier"> <intent-filter> <action android:name="com.yourpackage.HOURLY_CHECK" /> </intent-filter> </receiver> What are all these? You will see what HOURLY_CHECK is in a minute. The entries in the .StartupLauncher I gathered from multiple StackOverflow answers, claiming to work on multiple devices. What you are normally required to consume is just BOOT_COMPLETED, but put the rest just in case. Note the enabled attribute of the receiver – they are normally not needed, but make sure the recieve is not disabled by the parent application. Honestly, I wouldn’t advise for putting useless stuff “just in case”, but given the quirks of android devices, I’d keep them. Now implement the Service that will receive the BOOT_COMPLETED event: public class StartupLauncher extends BroadcastReceiver { @Override public void onReceive(final Context ctx, Intent intent) { AlarmManager alarmManager = (AlarmManager) ctx.getApplicationContext().getSystemService(Context.ALARM_SERVICE); Intent intent = new Intent("com.yourpackage.HOURLY_CHECK"); PendingIntent notificationIntent = PendingIntent.getBroadcast(ctx.getApplicationContext(), 1, intent, 0); alarmManager.setRepeating(AlarmManager.RTC_WAKEUP, 0, TimeUnit.MILLISECONDS.convert(60, TimeUnit.MINUTES), notificationIntent); } } Why not do it with a ScheduledExecutorService? Because once you schedule a Runnable, your BroadcastReceiver dies, and the executor dies with it, so the runnable is...
Recent Comments