I was pair-programming today and we made a change to a fundamental data type which required all the web-services to be rebuilt and republished.
We’d done this a couple of times before. The build is OK, but what then follows is a torterous process of manual FTPing 10 webserservices to a pre-live environment. Find folder locally, find folder remotely, select all..NOT web.config(!), double check you’re in the pre-live folder, copy, repeat.
The pragmatic part of me thought there had to be a better way of doing this. Why not just script it. Surely every man and his dog has a handy FTP upload script to recursively upload a folder of stuff?
Apparently not.
I tried PowerShell first. I like PS, but I’m a novice user trying to get better. Last time I used it (to find a string in a collection of files) I was blown away by how simple and elegant it was. 3 commands, piped together on one line, no fuss.
Turns out you can’t FTP in PowerShell.
OK, let me clarify… According to StackOverflow, you can, but only really by getting .NET to do it for you, and only then one file at a time. And the code…
# create the FtpWebRequest and configure it
$ftp = [System.Net.FtpWebRequest]::Create("ftp://localhost/me.png")
$ftp = [System.Net.FtpWebRequest]$ftp
$ftp.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftp.Credentials = new-object System.Net.NetworkCredential("anonymous","anonymous@localhost")
$ftp.UseBinary = $true
$ftp.UsePassive = $true
# read in the file to upload as a byte array
$content = gc -en byte c:\me.png
$ftp.ContentLength = $content.Length
# get the request stream, and write the bytes into it
$rs = $ftp.GetRequestStream()
$rs.Write($content, 0, $content.Length)
# be sure to clean up after ourselves
$rs.Close()
$rs.Dispose()
That’s not a solution.
You could also do it using NetCmdlets, some sort of add-on for PowerShell, but seriously, even if it is free for one concurrent connection, registering, providing email addresses, downloading, installing … just to put some FTP away? In a scripting language?!
Where else did we try? Oh, we tried NAnt as well – I thought they must have a decent <ftp> task. Apparently not. The only one anywhere close was this NAnt FTP Task – but don’t waste your time getting excited about the example because although it looks like it does exactly what we needed, the link is broken and there’s no trace of an alternative.
We tried DOS’ FTP mput as well … but this isn’t recursive.
We did think that it would be pretty cool if there was an FTP client which recorded your commands, allowing you to play them back later, or save them to a script file, but alas, such a client doesn’t exist. Add it to the list of programs I will one day get around to making (yeah, right).
By this point, things were getting pretty desperate, and we started to consider just writing a small .NET Console Application to automate the transfer for us, although that would have had its problems as well.
Finally, however, we came across WinSCP. At one level, this is just another FTP Client. However, you can run it in command line mode, and you can send scripts to it. And it’s clever enough to know that when you specify a folder instead of a file, it needs to copy everything in that folder.
Within 10 mins of downloading, we’d thrown together a simple script using WinSCP’s put command which worked perfectly. The GUI isn’t too shabby either, so I’m pretty sure that WinSCP will be replacing CoreFTP as my new FTP Client of choice. It’s supposed to be a SSH file copy client (Win SecureCoPy), but it does FTP pretty well too, and is a true open-source app under GPL.
WinSCP - more screenshots at http://winscp.net/eng/docs/screenshots
New Year is a good time for some introspection. I’ve been looking at how I can do things, and re-reading the most excellent book The Pragmatic Programmer: From Journeyman to Master.
One of the mantras of a pragmatic programmer is avoiding repetition by using automation.
In VB.NET, creating a new class file usually involves the same things again and again:
- Setting Option Explicit On
- Setting Option Strict On
- Importing some frequently used namespaces
- Adding some comments about who created the file, and why
The default template provided for VB.NET class files is pretty poor:
Public Class #CLASSNAME# End Class
Even if we can just add the Options in, it will save a couple of seconds each time. More importantly, it won’t ever be forgotten.
This is easily done in Visual Studio. Within the application folder structure, navigate to /Microsoft Visual Studio 10.0/Common7/IDE/ItemTemplates.
Where you go from here depends slightly on your language. For VB.NET it’s VisualBasic/Code/1033/Class.zip.
Copy and extract the .zip file, and simply edit the code file to contain whatever you want, then carefully zip back up and replace. MSDN has a good list of available template commands which you can use to insert the current user, date etc.
Finally, rebuild the templates for VS start devenv with the flag /installvstemplates
NOTE: don’t use the in-built Windows 7 ZIP File extensions to unzip/zip the zip file, as the file it creates isn’t recognised by Visual Studio, resulting in the modified template not showing at all in the list of types to add! Instead, use something better, like 7zip.
Here’s my new New Class file now:
The web is a big place. Sometimes though, it’s hard to visualise quite how big, and how much goes on in it.
Barry Ritholz has produced two amazing Infographics which really bring home quite how amazing the whole thing is. Every 60 seconds…
- 168 million emails are sent
- 695,000+ Facebook statuses are updated
- 70 new domain names are registered
What have YOU done in the last 60 seconds?
used with much thx from: http://questionmark.blogsome.com/
Wishing everyone a very Merry Christmas, whatever you’re doing, and a Happy New Year. 2011 has been a pretty good year for me, hopefully 2012 will be even better.
As you read this, just spend a moment imagining all the servers, switches, routers, stations, towers and other paraphernalia that need to be on and functioning to bring you this post over the net to wherever you are, whatever you’re reading this on. Heck, I’m writing this on Friday 23rd, and relying on geek-magic to publish it for me.
And then spare a thought for all the poor engineers, support desk workers and other geek-staff who have to work today to keep the whole thing working for you and running smoothly, just so you can check Twitter after your turkey.
Have a great Christmas, and see you in 2012.
Tom
This is my first post on Lync, if you don’t count the Who’s In? Who’s Out? post.
I’m soon going to working on Lync a lot, so I wanted to provide somewhere to send family and friends to answer the anticipated question: What the heck is Lync?
(by the way, if you don’t know the origin of the ‘What the heck is THAT?’ phrase, watch this: http://www.youtube.com/watch?v=sv4Hpz-GI3g. You’ll be glad you did.)
I found this video done by Microsoft which sort of explains what it is, without actually explaining what it is. That’s what I think anyway, which probably means it’s a good, non-technical overview of Lync, describing what problems it solves, rather than trying to explain how it solves them, or how it’s made up, or any of the technical things I try and get across to people, resulting in that glazed-over expression.
I think it’s exciting. I think the problems it solves are real, and I think they need solving. Hopefully that comes across in this video, and you can see why it’s cool.
If you’re technical, or just have trouble sleeping at night, you can see the keynote where Lync was officially launched. You can even see Bill Gates using it! Get the video direct from here, or see the Launch Website (for some reason the video’s don’t stream)
I’ve just launched a new website: PADI Medical.com
Here’s the history:
Last week I was conducting a Discover Scuba Diving session. I reviewed the forms at the start of the session and noticed that one of the applicants had marked ‘YES’ to the pneumothorax question. Obviously it meant no diving for him, and I explained this. I asked whether he’d been asked on the phone when booking about medical history. He had (the school I work with have a policy of asking people when the sign up if they have any “medical conditions” we should know about). He’d say no: but that was because his pneumothorax was 20 years ago, and he’d had no problems since then. In his mind, he had no medical conditions at all – but to us, he couldn’t dive (he got in the pool as his kids were also doing the session and swam with them, and I believe he’s coming back soon).
It occurred to me that we (as dive centres and Instructors) need an easy way to get people to look at the medical questionnaire before we see them in person for the first time. Obviously, the internet is a great medium, and in fact the dive school I work with does have a copy of the medical form. However, it’s buried deep in the site and hard to talk people through it on the phone, or give them a long link. It’s also potentially not a good marketing experience to link to the medical form from, say, a prominent place such as the home page as it has possibly negative connotations (it shouldn’t – diving is a lot safer on paper than, say, learning to snowboard, but perception is a powerful thing).
That’s why I created padimedical.com. It’s a free to use site that just lists the questions in the medical questionnaires. It’s an easy-to-remember URL that can be given over the phone or put in an email. It’s not tied to any particular dive centre or Instructor, so it could be used by anyone. At the moment it’s just in English, but I’ve plans to translate into the mainstream languages soon. I can’t promise that it will never have some subtle sponsored advertising on it, but the aim is to avoid that disappointment that I experienced last week when I told someone they couldn’t go diving, whereas if they’d seen the medical list beforehand they could have booked a physician appointment in advance.
PADI Instructors and Dive Centres: feel free to use the URL, pass it out to customers etc.
I’ll admit, before we started this, I didn’t even know BizTalk had a Throttle. I suppose it makes sense, it’s a feature of an expensive enterprise service bus that it’s “clever” in knowing about the state of resources around it and reacting accordingly.
Here’s what Microsoft have to say about BizTalk Throttling (from MSDN):
The BizTalk Server host throttling mechanism continually monitors for a throttling condition, calculates the severity of the throttling condition, and applies host throttling progressively depending on the calculated severity. The throttling mechanism is self tuning and the default configuration options are suitable for the majority of BizTalk Server processing scenarios. BizTalk Server host throttling exposes several configurable options that can be used to tune throttling for specific scenarios.
If you don’t know about it, it can appear that BizTalk is hanging, running slowly, or in some other way underperforming. You might even start to consider whether it is capable of handling the level of traffic you’re giving it.
It turns out that it almost certainly is capable, but for one reason or another it’s decided to slow itself down. When this happens (or you think it has happened), you need to know why: as you have a performance bottleneck somewhere in your system that at worst you need to know about, and at best, iron out.
There are loads of useful perfmon counters for BizTalk, but I’m going to concentrate on the following ones:
- BizTalk:MessageAgent
- DatabaseSize
- Message delivery incoming rate
- Message delivery outgoing rate
- Message delivery throttling state
- Message delivery throttling state duration
- Message publishing delay (ms)
- Message delivery delay (ms)
- BizTalk:MessageBox:General Counters
- Host Queue – Length
These are the ones I’m using at the moment, but there are plenty of others that can help you work out what’s going on.
The Message delivery throttling state is a good one to start with. Rather than being a scalar representation of some rate or value, this is one of a set of (I think) 7 values:
- 0: Not throttling
- 1: Throttling due to imbalanced message delivery rate (input rate exceeds output rate)
- 3: Throttling due to high in-process message count
- 4: Throttling due to process memory pressure
- 5: Throttling due to system memory pressure
- 9: Throttling due to high thread count
- 10: Throttling due to user override on delivery
So, if the value is 0 then no throttling is taking place. The Message delivery throttling state duration counter shows you how long (in seconds) BizTalk has been in the current throttling state. So, a well-running, happy system might look like this:
Any value other than 0 indicates some sort of throttling.
I don’t (personally) think that throttling always indicates a problem with a BizTalk setup or implementation. For instance, temporary throttling due to an imbalanced message delivery rate which lasts for a small amount of time and then subsides is just a good example of throttling stopping a problem from getting worse. Or it might be that BizTalk is catering for a bottleneck further down the line (for instance a slow receiving host) and is managing system flow to prevent problems.
However, some of the memory throttling states probably warrant some investigation if they are happening frequently.
When throttling is active, BizTalk literally slows down the rate at which it processes messages. You can see this happening by monitoring the Message publishing delay (ms) and Message processing delay (ms) counters.
Altering Throttling Conditions
You can adjust how, why and when BizTalk throttles, away from the default settings. I think you should be confident that you have a particularly unusual setup, or a good knowledge of BizTalk before going down this route: the defaults are presumably the result of lots of research and should probably be respected.
Rate-Based Throttling Settings
Right click the Host Instance and select Settings. Click the Host tab. There are 3 different tabs, for Resource-based, Rate-Based and Orchestration Throttling. Roughly: rate-based throttling is applied if there is a mismatch between the incoming and outgoing message speeds, resource-based identifies problems with threads, memory and database size, and orchestration-based allows you to define when/if dehydration occurs and subscription pausing/resuming. Each tab has specific sliders to alter how the throttling engine performs.
I’m the very first to admit I’m not the authority on BizTalk and it’s Throttling Engine, so here’s some of the MSDN pages I’ve found useful in understanding, identifiying and diagnosing our own particular bottleneck problem, which manifested itself initially in BizTalk throttling:
MSDN: How BizTalk Server Implements Host Throttling
Disclaimer: this is a non-IT related rant. If it has any bearing on software development at all, it is that it is further validation of the Doing Things Right First Time pattern.
We moved house a few months ago and since then I’ve been trying to do small things to improve it, things I can do myself and which don’t cost a lot of money. For some reason, one of the bedrooms at the back of the house was considerably colder than the others, so I went up to the loft to investigate.
The loft space is in two parts, reflecting the additional work done on the house in the past. The main part is original1890′s: high pitched roof, plenty of space. This area already had lots of insulation on the floor, some of which I’d recently upgraded when I part-boarded it for storage.
The second part adjoins the main part, and is a 1960′s extension. Because it joins onto the side, the rafters from the main roof come all the way down, so the only access to this new part is through a gap at one end. The dimensions of this gap are about half a metre squared. In case you’re having trouble imagining that, it’s slightly narrower than my shoulders. In fact, it’s also slightly narrower than my hips as well.
Shining a torch through the gap, I saw the beams holding the ceiling up, and absolutely no insulation.
Now, why on earth would you choose to add an extension anytime in the last 50 years, and then NOT add insulation? Is loft insulation a modern phenomena, younger than ARPAnet and what we now call the Internet? I’m not really up there on my history of loft insulation, but according to at least one source, in the 1950-60′s there was a minimum standard for loft insulation of 50mm. So, our Norwich-based builder of yesteryear was both silly AND illegal?
I’d already bought some loft insulation – the sort that can come apart into two layers to allow you to put a thinner layer between the joists. Luckily it was already pre-cut into small sections of about a foot width, so it was relatively easy to roll it up and shove it through the gap. All I had to do now was follow it through.
This was probably the worst part: going through for the first time. I had two torches to give some light (the single bulb in the main loft did nothing in this corner). I’d bought some extra loft boarding which I laid in front of the gap. This allowed me to lie on my font facing the gap. By twisting my body and pulling myself through I managed to get my shoulders the other side, and I could turn around again, back onto my front, wedged between the rafters.
I pulled myself forward until my hips hit the rafters, then repeated the process. The further forward I went, the harder it got, because I had to be careful to only support myself on the joists, so that I didn’t go straight through the ceiling. Finally I was through, although I was more worried about how I was going to get back out.
The maximum height of
this space was about a metre and a half which decreased to nothing at a 45 degree angle over about 2 metres. So, I had to stay at a crouch all the time. My original plan was to pull through one of the loft boards, but I gave up on that as a bad idea, opting instead to kneel on the joists.
The actual laying of the insulation was the easy part, more or less. I’d shoved through almost enough insulation to do all of the base layer (between the joists). In order to push it into the gaps at the ends where the roof met the floor, I had to lay right down again, supporting myself across 2 joists. Good for the stomach muscles, but I was convinced I was going to put a foot wrong and come through.
From this position, amongst the dust and fragments of mortar, I could see the blue British Gypsum logo of the plasterboard used for the ceiling. Well, that was good: I had pretty much resigned myself to finding the dusky off-white of asbestos there instead. Even the loss of one of the torches didn’t put me off now, the other one was still going strong.
Once I’d been in and come out once, it didn’t seem so bad. I had to keep going in and out to retrieve more insulation. The second layer is supposed to be laid at full, 200mm thickness. That means that I couldn’t keep it rolled out as it wouldn’t fit through the gap, I had to unroll it, then try and pull it through at an angle. Insulation is really good at sticking to everything, so each roll would fragment as I pulled it, meaning that I needed to go out and come in twice for each length.
Moving around reminded me of diving on an old and fragile wreck, performing a tricky penetration in silty water. You had to be aware of your surroundings in 3D (including behind and above you), plan every move and execute it slowly and deliberately. It was whilst I was delicately turning round 180 degrees to face the entrance gap to pull through another wad of insulation that the final torch battery gave out. That was when I decided to call it a night.
So, now both torches are charging up again, I’ve had a shower, and now a rant, and feel better for it. I’ll finish off tonight, hopefully avoid putting my foot through the ceiling, and that room had better warm up considerably and make the whole thing worth while!
I like when I find out something new, especially when it’s been under my nose for years and I never realised it. Sometimes I get scared about how little I know, in the grand scheme of all the things there are to know, but that usually spurs me on to continue reading, and learning.
Here’s a good example. I’m not a DBA but I’d say that I have a pretty good working knowledge of SQL. I think you need to if you develop data-bound applications. And, right now, we have a deadlock problem. Something, somewhere, is bumping heads with a new piece of code we’ve written. I wouldn’t mind, but our code is being chosen as the deadlock victim, so it looks like we’ve got it wrong.
So, how do work out what to do. Of course – to the Profiler!
The SQL Profiler is really useful, and I’ve used it for all sorts of things in the past: diagnosing integration issues, improving performance, even disassembling what 3rd party apps are doing behind the scenes. The only problem I have with it is that it chucks out a lot of data, if you don’t really know what you’re looking for. In our particular instance, that’s a Bad Thing. I’ve already found out the hard way that trying to run a Tuning trace for the Database Engine Tuning Advisor is just asking too much of the server at the moment. It shouldn’t be, but there we are.
So, not really knowing what to look for (apart from a Deadlock, somehow) and knowing that if I wanted to use the Profiler I’d have to be careful, I went searching for something to read. I found this: How to Track Down Deadlocks Using Sql Server 2005 Profiler. This is a really well-written, clear guide to using the Profiler to find locks. For instance, I’m only looking for 3 very specific types of messages to do with deadlocks at the moment, which means I can leave the trace on all day without it causing any performance issues.
Even better, when I eventually find them, the Profiler will tell me exactly what T-SQL was run, who was victim, who was succesful, and why. Graphically! I didn’t even know the Profiler did graphics!
SQL Deadlock (image from http://www.simple-talk.com)
The oval on the left (with the cross) is the victim, the one on the right was the SQL that ran. If you hover over the ovals you get the actual SQL commands issued. The resources are shown in the middle.
If you’re having deadlock issues and don’t know where to start, I highly recommend you go and read the article. Even if you don’t think you are having deadlock issues, it might be worth setting up a trace to look for them, just to make sure.
