by Bozho | Oct 15, 2013 | Aggregated, Developer tips
Until two years ago I was reading tech articles every day, I was regularly watching presentations, I was contributing significantly to stackoverflow. Then I gradually reduced my “efforts” of this sort. Followed a couple of tech aggregators (HN, reddit, DZone) on twitter, and that has been the only source of blogposts and articles ever since. I almost stopped answering stackoverflow questions. In a way, I ditched the habit of constantly and pro-actively engaging with the tech world, and started doing that on an on-demand basis, only occasionally reading, watching or answering. Why is that? Probably I began to think of myself as “expert”, and most of the stuff I stumbled upon was either already familiar to me, or too niche. Yet another web framework? Yet another article describing a best practice I already know. Yet another very specific usecase that I would only need to read when confronted with the problem, and then Google will find that article for me. Most of the questions on stackoverflow were old questions repeated. “How do I resolve this exception”, “How do I do this task with this framework”, “How to do X in Java”. And probably I was overwhelmed by the tons of information poured into my RSS reader every hour. I eventually stopped using RSS and switched to twitter, where I was getting just a fraction of the tech pieces I used to get with RSS. Is that bad? Am I feeling less up-to-date and therefore less capable of picking the right technique or technology for a given problem? I don’t think so, at least not yet. Is this the regular...
by Bozho | Sep 18, 2013 | Aggregated, Developer tips
As part of a project I’m working on, I needed to get documents from state institutions. And instead of writing code specific for each site, I decided to try creating a “universal” document scraper. It can be found as a separate module within the main project https://github.com/Glamdring/state-alerts/. The project is written in Scala, and can be used in any JVM project (provided you add a scala jar dependency). It is meant for scraping documents, rather than random data. It can probably be extended to do that, but for now I’d like it to be more (state)-document / open-data oriented, rather than a tool for commercial scraping (which is often frowned upon). It is now in a more or less stable form, I’ve already deployed the application and it works properly, so I’ll just share a short description of the functionality. The point is to be able to specify scraping only via configuration. The class used to configure individual scraping instances is ExtractorDescriptor. There you specify a number of things: Target URL, http method, body parameters (in case of POST). You can put a placeholder {x} which will be used for paging The type of document (PDF, doc, HTML) and the type of the scraping workflow – i.e. how is the document reached on the target page. There are 4 options, depending on whether there’s a separate details page, whether there’s only a table and where the link to the document is located XPath expressions for elements, containing meta data and the links to the documents. There’s a different expression depending on where the information is located – in a...
by Bozho | Sep 2, 2013 | Aggregated, Opinions
PaaS (Platfrom as a Service) is a marketing term for a sandboxed environment where applications can be deployed. The sandbox usually includes a lot of 3rd party software, which can be used by the application. Notable (java-related) PaaS’s include GoogleAppEngine, Heroku, OpenShift, CloudBees, CloudFoundry, Jelastic. It all sounds very nice – you configure your application via a web console, you include the services you need (application server, database, message queue, search engine), add a few configuration hooks and everything is up and running. No need to go and install stuff manually, no need to think of scaling, as this may be automatically handled by the platform – you simply add more instances/engines/dynos/gears/whatever to service the request. But I’ve evaluated a couple of them in the past years and encountered one serious flaw that was essentially a deal-breaker for most projects: software versions. During the development of each project, a part of the stack is upgraded to the latest versions. That’s due to either important bug fixes, or performance improvements, or for the sake of staying up-to-date. In virtually all projects I’ve used, we have found a need to upgrade one or more components. The message queue has a serious bug, then a NoSQL database needs some performance improvements from the new version that covers the specific case used in your project, then the search engine offers a nice distributed feature that you’d rather use. And yet, the managed environments rarely offer the latest versions (if they offer the 3rd party software at all). They might upgrade after a while, or they might not. You don’t have control over that....
by Bozho | Aug 18, 2013 | Aggregated, Opinions
We are all familiar with the KISS (Keep It Simple, Stupid) and YAGNI (You Ain’t Gonna Need It) principles. And yet, overly complicated software is everywhere. Let’s use an application server. Definitely. We can’t go without distributed transactions. And a message queue – for the sake of decoupling. Oh, our business logic is likely to include a lot of business rules, let’s go for a rule engine. And an ESB, definitely an ESB – how else will we distribute our multiple modules without making them interdependent? Database? Oracle, of course. But also CouchDB and Cassandra, because relational databases are not good at some things and we will need very high performance. And we need a couple of additional layers. Some facades here and there. And modules – we just shouldn’t have a monolithic application, how else will we be able to reuse our components? You know, let’s try OSGi, it will suit us. Did I forget web services? SOA is so important. And the ESB requires it anyway. Even if you don’t have designated architects, this is the sort of thing that happens. And each of the things has some seemingly legit reason to be there. No. YAGNI. You are most probably building business software that will be used by not more than a few hundred users simultaneously. It is probably going to include some business logic and a lot of requirements for reports and statistics. And you don’t need any of the above in order to make a functional, easy to maintain piece of software. Am I contradicting myslef when saying you should leave technologies and frameworks out,...
by Bozho | Aug 2, 2013 | Aggregated, Developer tips
We need to make multiple web-based projects with a lot of shared functionality. For that, some sort of a plugin system would be a good option (as an alternative to copy-pasting stuff). Some frameworks (like grails) have the option to make web plugins, but most don’t, so something custom-made is to be implemented. First, let’s define what is the required functionality. The “plugin”: should be included simply by importing via maven/ivy should register all classes (either automatically, or via a one-line configuration) in a dependency injection container, if one is used should be vertical – i.e. contain all files, from javascript, css and templates, through controllers, to service layer classes should not require complex configuration that needs to be copy-pasted from project to project should allow easy development and debugging without redeployment The java classes are put into a jar file and added to the lib directory, therefore to the classpath, so that’s the easy part. But we need to get the web resources extracted to the respective locations, where they can be used by the rest of the code. There are three general approaches to that: build-time extraction, runtime extraction and runtime loading from the classpath. The last approach would require a controller (or servlet) that loads the resources from the classpath (the respective jar), cache them, and serve them. That has a couple of significant drawbacks, one of which is that being in a jar, they can’t be easily replaced during development. Working with classpath resources is also tricky, as you don’t know the names of the files in advance. The other two approaches are very similar....
Recent Comments