CISCO---ATM problem case----CLOSED

夜游神

Problem Description:

1 Customer complain poor performance between two domestic sites.

Source: 172.31.224.35

Destination: 10.141.131.1


01R#ping i
Target IP address: 10.141.131.1
Repeat count [5]: 100
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface: 172.31.224.35
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.141.131.1, timeout is 2 seconds:
Packet sent with a source address of 172.31.224.35
!!!!!!!!!!!!!!!!!!!!!!!!.!.!!!!.!.!!!!!!!!!!!!!.!!!!!!!!!!..!!!!..!!!!
!!.!!!!!.!!!!!!!!!!.!!!!!!!.!!
Success rate is 87 percent (87/100), round-trip min/avg/max = 80/198/344 ms

2 There is a IMA link with 4 E1 ATM links bind for this WAN link. So Customer needs answer that why the traffic inbetween these two sites always around 6M bandwidth and cannot burst to 8M.

This is a very very common ATM BW issue regardless ISP transmition system, hardware WIC cards or configures ~~~
The reason is because of the overhead, or ATM "cell tax" which is a 5bit header overhead that is put on every packet.The minimum overhead under prime conditions is about 9% but can easily average at 12 to 15%.As every packet is taxed, if you are sending a lot of small packets, the overhead could be much higher. As such the max utilization in case of ATM is 80% of the link, in that sense in this case what this router is giving as per customer 6M bandwidth is absolutely perfect.


Please provide a troubleshooting solution and answer the customer's question.  请不要考虑hardware or IOS bug issue!!! 相关端口配置也正确！！

[ 本帖最后由 夜游神 于 2009-7-10 13:41 编辑 ]

degra

raise a tac case :)

austart

两年前处理同样的故障，链路丢包严重，后来发现是路由器上ATM模块,HSSI模块冲突啊，让思科开几个月CASE也没解决，只好把相关模块装到不同路由器了。

夜游神

原帖由 gandu 于 2009-7-10 11:43 发表
raise a tac case :)

Good Idea!!!

degra

无论cisco 还是juniper,  那些 x 口 PA 都发现过不少硬件兼容的问题 i.e pa-4t  -> pa-4t+

夜游神

原帖由 gandu 于 2009-7-10 12:30 发表
无论cisco 还是juniper,  那些 x 口 PA 都发现过不少硬件兼容的问题 i.e pa-4t  -> pa-4t+

This one could cause traffic drop or link flapping~~~Besides, raising a TAC ticket to Cisco and this one, do you have you own troubleshoot solution for both Q1&Q2?

degra

I think you need to ask you customer:
when the problem starts, if its a day one problem, I would look for design/configuration issues. otherwise was there any hardware/configuration change before problem starts; check for crc errors, are 4 links active in the ima bundle?,  bundle utilisation graph >24hrs, etc; show tech is good to start with  

and most importantly, what is the impact ! so you can set priority

夜游神

Follow your suggestion for Q1, I done relugar "show check" at IMA links and found CRC error free there, also confirm there is no clocking issue for this IMA group, what to do next?

Can you try to answer Q2 as well?

[ 本帖最后由 夜游神 于 2009-7-10 13:14 编辑 ]

hwll

开tac case之前, 找你的atm service provider吧

另外用 "sh ima int atmx/imay "看看是不是四条链路都是active的...6M听起来好像只用到3个2M

[ 本帖最后由 hwll 于 2009-7-10 13:28 编辑 ]

ios

if it is cisco 7206, could be a bandwidth point issue.

For Q1
add access-list on both ima interface and do the ping test. so that we could tell where the packet got dropped.

access-list 100 permit icmp echo
access-list 100 permit icmp echo-reply
access-list 100 per ip any any
access-list 110 permit icmp echo
access-list 110 permit icmp echo-reply
access-list 110 per ip any any
int imax/x.x
ip access-group 100 in
ip access-group 110 out

For Q2,
seems one of the E1 is faulty.
show interface to check counter
also check with your service provide

夜游神

原帖由 hwll 于 2009-7-10 13:26 发表
开tac case之前, 找你的atm service provider吧

另外用 "sh ima int atmx/imay "看看是不是四条链路都是active的...6M听起来好像只用到3个2M

Sad for your answer to my Q2~~~~~~

This is a very very common ATM BW issue regardless ISP transmition system, hardware WIC cards or configures ~~~
The reason is because of the overhead, or ATM "cell tax" which is a 5bit header overhead that is put on every packet.The minimum overhead under prime conditions is about 9% but can easily average at 12 to 15%.As every packet is taxed, if you are sending a lot of small packets, the overhead could be much higher. As such the max utilization in case of ATM is 80% of the link, in that sense in this case what this router is giving as per customer 6M bandwidth is absolutely perfect.

javed

LZ给的信息太少,别人又没有办法直接登到LZ的设备上去处check, 这种情况下请LZ先提供以下信息:

Q1:
1, ping 丢包是一直有的问题还是最近发生的?如果是最近发生的,有没有具体的Timestamp, 或者有没有相应的trigger?比如说哪里的链路断过啊,有新的业务加上去啊等等.

2, 这个问题是仅仅局限在这两个节点之间,还是别的节点,或者说172.31.224.35到别的目的地址也有问题? 或者是别的节点到10.141.131.1也有问题?

3, 这两个节点之间总共有几跳?有没有做过分段的ping,结果如何?是每段都有丢包,还是其中一段有丢包?

4, 是不是每次ping的结果都是87%?

Q2 (类似的提问):
1, 这是个一直有的问题还是最近发生的?如果是最近发生的,有没有具体的Timestamp, 或者有没有相应的trigger?

2, 这个问题只局限在这段IMA link上,还是别的IMA link上也有? 如果仅是这一段IMA link有问题,那它和其它IMA link的差别在哪里?

3, ATM的具体类别是什么? VBR-rt/VBR-nrt/CBR/UBR/etc? 每一种类别都有问题还是其中一种有问题?

4, 这段IMA 4 个E1 链路上的流量有什么区别? 都差不多还是其中一条E1和其他的不一样?

以上问题都是很标准化的问题(what/where/when/extend), 对于一个对LZ的网络环境毫无了解的人来说,这些信息很重要,可以帮助别人迅速narrow down问题所在,然后再做下一步分析......

hwll

原帖由 夜游神 于 2009-7-10 13:36 发表



Sad for your answer to my Q2~~~~~~

This is a very very common ATM BW issue regardless ISP transmition system, hardware WIC cards or configures ~~~
The reason is because of the overhead, or A ...

这要看你在哪里测的带宽利用率了... sh int atmx/imay 的结果是已经考虑了overhead的, 是sar处理后的结果..

javed

我还以为Q1和Q2是毫无关系的两个case呢, 你在Q1里的description里并没有讲这是个IP over ATM的连接啊?仅仅是说两个节点layer3 ping的问题.
如果我知道Q1也是通过IMA的话,我就不会问你分段ping的问题了.

同样,LZ对于Q2的描述,也没让我看出Customer所说的"Traffic"是IP traffic. 如果是纯粹的ATM traffic,我想就是完全另一回事情了.(和小包不小包无关了)

[ 本帖最后由 javed 于 2009-7-10 13:57 编辑 ]

夜游神

原帖由 javed 于 2009-7-10 13:48 发表
我还以为Q1和Q2是毫无关系的两个case呢, 你在Q1里的description里并没有讲这是个IP over ATM的连接啊?仅仅是说两个节点layer3 ping的问题.
如果我知道Q1也是通过IMA的话,我就不会问你分段ping的问题了.

OK啦，你的回答超级精彩的；每个人的答案也都很正点咧．都是刁人～～～本来TS就没有套招，你怎么理解问题，想按自己的方式去TS没有任何问题的：P
我提的问题也的确是缺乏很多info.~~~~

盒饭强

这是面试的题目吗?