does anyone know to where the New Zealand online callbook was moved?


Dave AA6YQ
 

I learned last night that Pathfinder's "NZ Callbook" search is no longer working...

73,

Dave, AA6YQ


Nigel ZL2DF
 

Hi Dave,
https://rrf.rsm.govt.nz/smart-web/smart/page/-smart/domain/licence/SearchCertificateAndIdentifierPage.wdk?showExit=Yes

I posted a similar comment 26/6/2019 relating to the 3 stage process but did not have any luck with a follow up to RSM on that occasion.

I asked

"On the page

https://rrf.rsm.govt.nz/smart-web/smart/page/-smart/domain/licence/SelectLicencePage.wdk there is a callsign field where the full callsign can be entered but unfortunately this does not return any information.        Could this field be activated so as to return the holder’s details?       Do you have any other suggestions on how the data may be accessed using the full callsign?"

Am willing to try approaching them again.     Any suggestions on what would be useful to ask for?

 

73,
Nigel
ZL2DF


Dave AA6YQ
 

+ AA6QY comments below

https://rrf.rsm.govt.nz/smart-web/smart/page/-smart/domain/licence/SearchCertificateAndIdentifierPage.wdk?showExit=Yes

I posted a similar comment 26/6/2019 relating to the 3 stage process but did not have any luck with a follow up to RSM on that occasion.

I asked


"On the page

https://rrf.rsm.govt.nz/smart-web/smart/page/-smart/domain/licence/SelectLicencePage.wdk there is a callsign field where the full callsign can be entered but unfortunately this does not return any information. Could this field be activated so as to return the holder’s details? Do you have any other suggestions on how the data may be accessed using the full callsign?"

Am willing to try approaching them again. Any suggestions on what would be useful to ask for?

+ Given that the web interaction is encrypted, I cannot reverse engineer the parameters and values to be included with the invocation to cause information for a desired callsign to be displayed. Thus I would need documentation describing the parameters and values.

+ Previously, for example, directing the online callbook to display results for your callsign meant sending this HTTP POST:

http://www.happy.geek.nz/cgi-bin/nzartsmartquery.py?callsign= ZL2DF&Search+SMART=Submit

+ I determined this by capturing the HTTP transaction generated by a web browser invoking the above URL - but if I try that with

https://rrf.rsm.govt.nz/smart-web/smart/page/-smart/domain/licence/SearchCertificateAndIdentifierPage.wdk?showExit=Yes

+ the result is gibberish due to HTTPS encruption.


+ Today, pasting

http://www.happy.geek.nz/cgi-bin/nzartsmartquery.py?callsign= ZL2DF&Search+SMART=Submit

into a web browser's address bar yields this result:

Unexpected error: list index out of range

+ which implies that nzartsmartquery.py is still present, but unhappy with the parameters and values that previously worked. If you can determine who's responsible for

http://www.happy.geek.nz/cgi-bin/nzartsmartquery.py

+ please ask them what would produce the

Unexpected error: list index out of range

+ error message.

73,

Dae, AA6YQ


Nigel ZL2DF
 

Dave,
I have been following this up with the NZART Administration Officer.  (NZART is NZ's ARRL)     He points out that your search query is not directed to the Government Radio Spectrum Management (RSM) site but to a NZ ham's site that of ZL3AME.       This site appears to have been inactive since about 2008.      I have not been able to find any contact information for ZL3AME to confirm this

The correct site is 
Search the Register of Radio Frequencies (RRF) | Radio Spectrum Management New Zealand (rsm.govt.nz)

Could you look here and see if you can extract the information you need.      I have also been told that RSM are in the process of changing to a completely new database but when is uncertain.

73,
Nigel  ZL2DF


Dave AA6YQ
 

+ AA6YQ comments below

I have been following this up with the NZART Administration Officer. (NZART is NZ's ARRL) He points out that your search query is not directed to the Government Radio Spectrum Management (RSM) site but to a NZ ham's site that of ZL3AME. This site appears to have been inactive since about 2008. I have not been able to find any contact information for ZL3AME to confirm this

The correct site is
Search the Register of Radio Frequencies (RRF) | Radio Spectrum Management New Zealand (rsm.govt.nz) <https://www.rsm.govt.nz/licensing/how-do-i/use-the-rrf/search-the-rrf/>

Could you look here and see if you can extract the information you need.

+ Thanks, Nigel, but that documentation explains how a user would access the information with a web browser; it does not explain how an application like Pathfinder can emulate a web browser by providing the required search parameters in the required order. For the evidently now defunct ZL3AME callbook, Pathfinder sent this "post data"

callsign={TargetCallsign}&Search+SMART=Submit

+ to this URL:

http://www.happy.geek.nz/cgi-bin/nzartsmartquery.py

+ In the past, I reverse engineered the required data by using an application that captured the HTTP transaction when I manually queried the source using a web browser. The near-universal use of HTTPS - "secure HTTP" - makes that reverse engineering impossible, since the transactions are all encrypted. I can only develop new Pathfinder searches of HTTPS sites if the site owner documents required search parameters so that I can enable Pathfinder to issue them.

73,

Dave, AA6YQ


Nigel ZL2DF
 

I will persevere with my NZART contact.     Unfortunately government departments are not known for being generous with favours or free information.

73,

Nigel, ZL2DF


Dave AA6YQ
 

+ AA6YQ comments below
I will persevere with my NZART contact.
+ Thanks, Nigel.

       73,

              Dave, AA6YQ


David Westbrook
 

+ In the past, I reverse engineered the required data by using an application that captured the HTTP transaction when I manually queried the source using a web browser. The near-universal use of HTTPS - "secure HTTP" - makes that reverse engineering impossible, since the transactions are all encrypted. I can only develop new Pathfinder searches of HTTPS sites if the site owner documents required search parameters so that I can enable Pathfinder to issue them.

The encryption makes it impossible to reverse engineer if capturing the traffic in the middle of the network path,  but the browser devtools let you see exactly what the end client is submitting, and can determine from that how to construct the expected POST form payload.

While this callbook still may not be suitable for Pathfinder application, I was able to successfully write a script to lookup a callsign in the ZL callbook.  Unfortunately the form parameters are dynamically generated every page load, so there is not a static post data format  for pathfinder;  also there is not a single callsign field -- the prefix & suffix are separate fields in the form.

But in case it helps someone for reference, the methodology and code is below.  e.g. someone could write a little proxy server so that a simple url scheme like http://zl_callbook.example/com?callsign=zl2df   could be used, and it would do the needed logic & parsing below, and return the result.

I started at https://rrf.rsm.govt.nz/smart-web/smart/page/-smart/domain/licence/SelectLicencePage.wdk  to try and find ZL2DF, with no luck.
On that page, there's a nav item for "Search Register" -> "Search certificates and Callsigns"  and that worked! I could find ZL2DF.

Despite it being a HTTPS site,  I could still use browser devtools to capture the URL & payload.   When doing a search, the POST looks like this -- you can see the callsign fragments in the "S2175=ZL&S2178=FOLLOWED_BY&S2181=2DF" portion:

  -H "Connection: keep-alive" ^
  -H "Pragma: no-cache" ^
  -H "Cache-Control: no-cache" ^
  -H "sec-ch-ua: ^\^" Not A;Brand^\^";v=^\^"99^\^", ^\^"Chromium^\^";v=^\^"90^\^", ^\^"Google Chrome^\^";v=^\^"90^\^"" ^
  -H "sec-ch-ua-mobile: ?0" ^
  -H "Upgrade-Insecure-Requests: 1" ^
  -H "Origin: https://rrf.rsm.govt.nz" ^
  -H "Content-Type: application/x-www-form-urlencoded" ^
  -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" ^
  -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" ^
  -H "Sec-Fetch-Site: same-origin" ^
  -H "Sec-Fetch-Mode: navigate" ^
  -H "Sec-Fetch-User: ?1" ^
  -H "Sec-Fetch-Dest: document" ^
  -H "Accept-Language: en-US,en;q=0.9" ^
  -H "Cookie: JSESSIONID=J9XnzGOwoo85BGIINSKb69KztNOTaCsIc7v_wlPXhsSVMDezNpnh^!-1079729315^!1572164037; mf5=1290567109; _ga=GA1.3.140114273.1623091604; _gid=GA1.3.823959159.1623091604; visid_incap_1962811=x8WvtjTeRGy2c3BwMcd2edNqvmAAAAAAQUIPAAAAAAA0pUsAh03X6tRgob6pjT4Y; incap_ses_702_1962811=WCVXLkXHUS/qrFcCugG+CdRqvmAAAAAATQttjr42VYtDZf9fifF29A==; _gat=1" ^
  --data-raw "medClientHeight=&medClientWidth=&S2142=true&licenseeComponent=&S2154=&S2175=ZL&S2178=FOLLOWED_BY&S2181=2DF&S2205=CONTAINS&S2208=&S2220=STARTS_WITH&S2223=&locationComponent=&S2241=&S2262=&S2268=&S2118=0&S2121=0&p_access_no=&internal.wdk.wdkCommand=S2370&internal.wdk.wdkCommandArgument=&SMART_WDK_PAGE_CLASS=2XuReF7vs^%^2BbNOWTghTpzlYS2LjoiH9WkrVIjDjZ4XCCckpOWw8QyFfCJ2vsBI4dtWgwKqpg^%^2BBefpkJlY" ^
  --compressed

Inspecting the Response content,  can see the search results in a HTML table:

            <tr class="listEvenRow">
             <td class="">
              <a id="S32524" title="137786" onclick="dispatch('S32524');return false;" href="javascript:void(0);">137786</a>
             </td>
             <td class="">Mr N M HAYTON</td>
             <td class="">ZL2DF</td>
             <td class="">2857</td>
             <td class="">GAOC</td>
             <td class="">&nbsp;</td>
             <td class="">&nbsp;</td>
             <td class="">&nbsp;</td>
             <td class="">
              <a id="S32551" title="Amateur GURL" href="https://rsm.govt.nz/licensing/frequencies-anyone-can-use/amateur-radio-operators" target="_blank">Amateur GURL</a>
             </td>
            </tr>



then I headed to a long-time go-to personal favorite tool,  perl  and the WWW::Mechanize  module, which is a programmatic web browser (HTTP client).  Some trial and error, and looking at the HTML/javascript source of that callsign search page to figure out what it does on submit,  and I was able to get it to work;  Parsing the html for the search results name then yields the hits.    In appears that the search tool does a "starts with" method of matching,  so can get multiple results, but that could be filtered to the original request.

Here is the command-line POC in action:

$ ./zl.pl zl2df
$VAR1 = {
          'Callsign' => 'ZL2DF',
          'Name' => 'Mr N M HAYTON'
        };
$ ./zl.pl zl2dx
$VAR1 = {
          'Callsign' => 'ZL2DX',
          'Name' => 'Mr C C HANNAGAN'
        };
$VAR1 = {
          'Callsign' => 'ZL2DXR',
          'Name' => ' DAVID HINDSON'
        };

And the perl source code:

#!/usr/bin/perl

use strict;
use warnings;
use HTTP::Cookies;
use WWW::Mechanize;
use Data::Dumper;
$|=1;

my $callsign = shift @ARGV or die 'need callsign, e.g. ZL2DF';

my ($pfx, $sfx) = uc($callsign) =~ /^(ZL|ZK|ZM)(\d.+)$/;

my $cookie_jar = HTTP::Cookies->new;    # in-memory cookie jar
my $mech       = WWW::Mechanize->new( cookie_jar => $cookie_jar );


my @inputs = $mech->current_form()->inputs;
foreach my $i ( 0 .. $#inputs ){
  my $v = join ' ', $inputs[$i]->possible_values;
  # Find the prefix dropbox -- it's the element with ZK, ZL, ZM as possible options.
  if ( $v =~ /\bZK\b/ && $v =~ /\bZL\b/ && $v =~ /\bZM\b/ ){
    $inputs[$i]->value($pfx);                   # Set the search prefix we want
    $inputs[$i+1]->value('FOLLOWED_BY');        # the next consectutive form element is the "Followed By"/"Contains" dropbox
    $inputs[$i+2]->value($sfx);                 # and the next consectutive form element is the search suffix.
    last;                                       # done setting callsign values
  }
}


# the "submit" isn't a form element button -- it's a link.
# Need to emulate javascript that takes the "Search" link id, and sets it in a form field.
my $link = $mech->find_link( text => 'Search' );
my ($submitId) = $link->attrs->{onclick} =~ /'(.+)'/;
$mech->submit_form(with_fields=>{ 'internal.wdk.wdkCommand'=>$submitId });

use HTML::TableExtract;
my $te = HTML::TableExtract->new( headers => [ qr/Name/, qr/Callsign/ ] );
$te->parse( $mech->content );
my $table = $te->first_table_found;
my @c = map { s/^\s+|\s+$//g; $_ } $table->hrow;
foreach my $row ($table->rows) {
  my %h;
  @h{@c} = @$row;
warn Dumper \%h;
}

73!
--david
K2DW (ex-KJ4IZW)


On Sun, Jun 13, 2021 at 8:35 PM Dave AA6YQ <aa6yq@...> wrote:
+ AA6YQ comments below

I have been following this up with the NZART Administration Officer.  (NZART is NZ's ARRL)     He points out that your search query is not directed to the Government Radio Spectrum Management (RSM) site but to a NZ ham's site that of ZL3AME.       This site appears to have been inactive since about 2008.      I have not been able to find any contact information for ZL3AME to confirm this

The correct site is
Search the Register of Radio Frequencies (RRF) | Radio Spectrum Management New Zealand (rsm.govt.nz) <https://www.rsm.govt.nz/licensing/how-do-i/use-the-rrf/search-the-rrf/>

Could you look here and see if you can extract the information you need. 

+ Thanks, Nigel, but that documentation explains how a user would access the information with a web browser; it does not explain how an application like Pathfinder can emulate a web browser by providing the required search parameters in the required order. For the evidently now defunct ZL3AME callbook, Pathfinder sent this "post data"

callsign={TargetCallsign}&Search+SMART=Submit

+ to this URL:

http://www.happy.geek.nz/cgi-bin/nzartsmartquery.py

+ In the past, I reverse engineered the required data by using an application that captured the HTTP transaction when I manually queried the source using a web browser. The near-universal use of HTTPS - "secure HTTP" - makes that reverse engineering impossible, since the transactions are all encrypted. I can only develop new Pathfinder searches of HTTPS sites if the site owner documents required search parameters so that I can enable Pathfinder to issue them.

       73,

                Dave, AA6YQ










Dave AA6YQ
 

Many thanks, David; I hadn't though of using the browser's devtools to see the unencrypted transaction. I'd happy add a "split the callsign" capability to deal wit the ZL site, but if its post parameters are dynamic, there's no point.

Your recommended technique will likely be useful in discovering updated search parameters for other online sources of QSL information.

        73,

                Dave, AA6YQ