Aug 172011
 

This is a small utility I wrote in the process of my work. It is a bit ugly and nonetheless non-elegant and non-reliable, because the actual CPU utilization it measures for the switch also has the CPU needed to execute the “show system cpu” command from switch side.

Will be much more elegant if the switch reports the data to SNMP trap collector, but anyway. It will help you help yourself write your own script for this. This one is made to monitor the CPU while you work on other tasks.

The utility uses Tcl:Expect, Tcl:Math and Tcl:Tk. All of them are included in TclLibs package of your chosen distribution. The code is set to use SSH to the switch, but easily can be modified to work also with Telnet. Just un-comment the corresponding lines and set the “proto” value to be “telnet”. The other important values are ip, user, pass, enable, enablepass and the actual command the switch needs to show the CPU utilization. You can also tweak values of Delay and Width. I use 1000 and  600, because those are 10 minutes in one band-graph 100 pixels high.

CPU Utilization

CPU Utilization

Few disclaimers.

  • The code is not very good and will raise exceptions if the switch does not respond in time.
  • The code assumes that the result line will end in 1 to 3 digits followed by %-sign. (e.g. the result must say “current cpu is 100%” or “CPU 33%”)
  • There is virtually no guarantee the code will show you accurate values. It is for monitoring only.
  • When 10 minutes pass, the bar graph will start to scroll from right to left. No data is kept anywhere.
  • Part of this code is copy&pasted verbatim from the Tcl forums and discussions

 Posted by at 3:25 pm
Jul 212011
 

There are really no key requirements for you to be hired as QA in any company. At least I’ve never worked as one before but was hired for my history as system administrator and developer.
I have also done some tests in the past that were needed for my work, but have never seen a real test plan and have never written one or followed one. In my past work, everything we write was tested by ourselves. It was not a separate job, but was part of the work process. You write, you test.(If you have not read the previous 3 articles, you may find part 1 describing the basics of the QA workflow, part 2 getting some details on the ways of How to test as QA and part 3 describing few details of the actual methods of testing. In all cases, those methods may vary a bit in the company that will hire you in their ranks.)
The job description and requirements themselves, may surprise you. That’s what was written in the Job Advertisement I saw when applying:


Telco Systems (a subsidiary of BATM Advanced Communications) is a well-known manufacturer of a wide range of switches and routers destined for telecommunications and corporate networks. BATM specializes in the design and implementation of network equipment for IP switching and routing, and provides means for high-speed data transfer over optical and copper media. The company offers products with highest throughput, reliability and quality along with technical support for broadband transmission of video, voice and digital data over IP.

Telco Systems is looking to recruit a QA (Software Quality Assurance Engineer).

As part of an international team of software engineers you will:

  • Design, develop and perform testing procedures for verification of state-of-the-art embedded systems software for a range of network applications;
  • Maintain detailed testing documentation;
  • Be involved in the full implementation cycle of specific modules in Telco’s integrated whole-range-of-platforms networking operating system
  • Help Software developer teams in testing of comprehensive functionality
  • Help Technical support teams in initial on-site installation of new or enhanced modules
  • Maintain sophisticated knowledge in relevant networking standards, be part of the new features design brainstorming sessions

Requirements:

  • Willingness to develop testing procedures
  • Strong analytical abilities and troubleshooting skills
  • Experience in Unix/Linux environment
  • CCNA finished course is an advantage
  • QA experience is an advantage
  • Experience in networking is an advantage
  • Experience in script languages is an advantage
  • Fluent in English

Benefits:

  • A challenging and dynamic area of work involving competition against world’s leaders in networking
  • Professional environment, working in a team of qualified engineers;
  • Competitive remuneration package;

For most of Its part, this job advertisement, should not scare you. You don’t have to start from zero. In most cases someone has already made templates about everything you need to do and there will be someone to mentor over you and provide help until you are ready to go delve deeper. When I started learning what is required of me, there was already a big data base with test plans and detailed test cases. I only need to build over them. Testing documentation was already above 1700 pages and small work was required to keep it up to date only. The requirements were only guidelines of what a person should know to apply for this job, but were not all “must haves”. If a person is a good in what he/she does. No CCNA and no specific certificates are really needed. Only a desire to work. (one of my female colleagues does not even speak English well, but she is good technician and even English fluency is not 100% required in such cases).

Any experience in the field of the company products or relevant may be more than enough to start working as QA specialist in such company.

The salaries vary per country. The best payment goes in USA, ranging from 5 to 6 digits in year, while in India the salary varies from 100 to 600k rupees (14-to-100k EUR or 20-to-150k USD). It is normal for underdeveloped or developing countries the salary to be low. It is also normal for those countries to have LOTS of outsourced big companies. It is a shame, that in Bulgaria, those digits don’t even go Half-that-good. Our QA engineers are happy if they make 5 digits/year in EUR.

But I got away with this, and forget the important aspects of this work:

  • It is interesting!
  • It pays in another way – you learn new stuff.
  • It makes your mind very able to analyze problems.
  • It is not hard to do.
  • Takes about 6 months to learn, but is like the bicycle and swimming. You can’t forget It.
  • Gives you a firm grasp on some white hat techniques you can’t learn by normal means.
  • Helps you learn some script programming in the process.
  • Gives inner details of company products you can’t find in the WWW.
  • It is actually quite fun to break through someone else’s code and find nasty bugs.
  • Allows you to break it again after the developer decides “it’s working”.
  • Creates firm connections with 3 categories of IT specialists that can come handy in personal aspect if you want to evolve. Developers, System administrators and Technical writers.

So, IF you think this work is for you, go to your local Job market and apply to the 10 most promising QA entries there. I will try to help you with advices and ready test scripts in my blog.

Keep in touch 😉

 Posted by at 11:40 am
Jun 272011
 

So to summarize what is important for a SQA engineering team to have in mind while testing (read the first two parts here and here if you have not already):

  • Installation of the feature/product: No conflicts should be seen on a system level. Environment variables of the feature should not conflict other features.
  • Data complatibility between features: No integer variable should be sent by the feature to a process that expects a string of characters.
  • Feature compatibility: If a feature is expected to run after another feature has started, this check should be made in the begining.
  • Error generation: If a feature dies and noone can hear it screaming, the whole system depending on this feature can still die. Thus a non-critical bug for the feature can lead to a system crashing, which is quite fatal.
    • Strength of the code for this feature. If it breaks easily, and its existence is crucial, the whole system can break. Remember: A chain is as strong as its weakest link!
    • Performance of the feature. Find the possible bottlenecks before the customer finds them. It is better to have a well documented and tested product limitation in the specifications instead of an angry customer or a furious manager.
    • Recovery. If the feature breaks, can It restart and recover so it keeps working as best as possible with the scavenged data?
    • Backward compatibility: Can the system work with another system with older version of this feature?
    • Sensitivity. What happens if the feature receives non expected data from another feature and there is no failsafe mechanism?

    Those are few rules of thumb you should have while working as SQA engineer. Most of them are written in the era of software development but are still quite valid and important.

    Running all those tests on every minor or major fix is quite time consuming and but be automated nowadays. The repetitive testing for a specific feature on every major or minor patch is called

    Regression testing.

    This kind of testing is done to check if the system is still sane after the newly compiled release and most of the test categories can be skipped.

    It is important though to not miss testing the basics are working. So after the installation, data and feature compatibilities are automatically tested and some important tests as parameters check have the results, we go to the real regression tests. They often go without a plan and are an analysis of the patched bugs and their interoperability with old fixes:

    • What if the new fixes are not working?
    • What if the new fix opens new bug?
    • What if the new fix breaks old fix?

    This kind of testing is important if the final product is to be present without old bugs reappearing in the code, while we think they are already fixed. While this kind of test builds confidence in the feature stability in different versions, It is too much time consuming and can drive the attention away from real new bugs that can make it through blind spots of the whole test plan, so most of the regression testing is human-observing-automated. And while it is time-consuming and can’t catch all the bugs – it provides the needed confidence when testing new growing features in their development cycle. Most of those tests are set on a single system and can’t catch the big bugs in fully operational environment. That’s why there is

    Stress testing.

    What is important to test on a feature that is supposed to work on a heavily populated server or a high traffic network?

    • Learn its boundaries and try to overwhelm them. e.g. Try to configure 4095 VLANs on a single port and flood all of them with traffic generator.
    • Try to overflow a buffer. e.g. Try 1 million administrative logins to a device under testing.
    • Try to flood the feature with enormous volume of data and see if the feature survives.
    • Try how the feature will operate in spartan conditions like low Memory or high CPU.

    While those tests are very importand, the test cases involved with them often catch one single bug in their whole life since the cases were originally written. That’s why we often do the so called

    Exploration/exploitation testing.

    We know the product, we know the feature, we know the code, we know EVERYTHING. That exactly is the bad part of our testing. The customer knows nothing except the configuration example in the manual. He needs the units for actual work, not for lab testing. So he/she starts building a big mesh-network of units and starts wiring them with cables and sets routing protocols. What happens? You guessed right – we missed some major blind spot. That’s why the exploitation testing is also very important.

    • Often unknown bugs are found in other feature while we test some new feature. It’s important the person responsible for this other feature to be alarmed and a new test case to be added in his test plan so next time this blind spot missed by him to be retested.
    • Sometimes the forum of your competitors or a news group can have some interesting bugs you can try to reproduce on the software or equipment you are developing.
    • And it happens, when you test the feature again and again on every major release – the developers got some old buggy code pasted in the new feature so everything breaks. You should be vigilant about what and where enters as new feature and what possible break points it can have. If you are not aware – than you probably will not like the job at all.

    The job description itself, may surprise you.

     

     Posted by at 8:41 pm